Jul 28 17:07:09.448409 ip-10-0-129-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:07:09.448420 ip-10-0-129-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:07:09.448445 ip-10-0-129-82 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:07:09.448659 ip-10-0-129-82 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:07:20.601763 ip-10-0-129-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:07:20.601777 ip-10-0-129-82 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f989befec78448bb8f8c2f63c8ed0465 --
Jul 28 17:09:16.943350 ip-10-0-129-82 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:09:17.345623 ip-10-0-129-82 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:17.345623 ip-10-0-129-82 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:09:17.345623 ip-10-0-129-82 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:17.345623 ip-10-0-129-82 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:09:17.345623 ip-10-0-129-82 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:17.346666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.346525    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:09:17.350456 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350435    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.350456 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350452    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.350456 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350457    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.350456 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350462    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350467    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350472    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350475    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350479    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350483    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350488    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350491    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350495    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350499    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350503    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350507    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350512    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350516    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350520    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350525    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350529    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350533    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350537    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350542    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.350697 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350546    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350551    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350556    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350560    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350564    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350568    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350572    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350576    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350581    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350585    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350590    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350594    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350598    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350602    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350607    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350612    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350616    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350620    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350624    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.351546 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350628    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350633    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350637    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350642    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350646    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350651    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350655    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350659    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350663    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350667    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350672    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350676    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350681    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350685    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350689    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350695    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350702    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350708    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350713    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350717    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.352438 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350721    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350726    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350731    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350735    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350740    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350745    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350750    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350754    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350759    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350763    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350770    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350775    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350784    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350790    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350794    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350799    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350803    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350807    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350811    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.353282 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350815    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350819    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350824    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350828    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350832    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350837    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350841    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350845    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350849    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350853    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350857    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350862    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350866    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350870    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350877    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350882    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350887    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.350891    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352098    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352108    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.354111 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352112    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352117    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352122    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352127    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352131    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352136    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352141    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352147    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352152    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352157    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352161    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352165    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352169    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352174    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352178    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352182    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352186    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352191    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352196    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352200    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.354866 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352204    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352208    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352213    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352217    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352221    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352225    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352230    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352234    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352238    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352242    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352247    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352251    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352255    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352259    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352294    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352300    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352306    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352311    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352316    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.355398 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352321    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352327    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352332    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352336    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352341    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352345    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352350    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352355    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352359    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352365    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352369    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352373    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352378    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352382    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352388    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352392    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352397    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352401    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352406    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352410    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.355891 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352414    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352419    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352423    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352428    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352434    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352440    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352444    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352448    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352452    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352456    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352460    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352464    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352468    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352472    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352477    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352484    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352490    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352495    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352500    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.356393 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352505    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352511    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352516    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352520    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352524    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352529    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352535    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352542    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352546    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352551    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352556    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352560    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352565    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352570    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352575    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352580    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352584    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352588    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.352593    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353222    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.356837 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353231    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353235    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353239    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353243    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353249    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353253    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353257    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353262    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353267    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353271    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353275    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353279    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353283    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353287    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353291    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353296    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353301    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353305    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353309    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353313    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.357343 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353318    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353322    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353326    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353331    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353335    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353339    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353343    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353347    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353352    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353356    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353361    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353365    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353369    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353373    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353378    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353382    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353386    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353390    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353395    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353399    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.357820 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353404    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353408    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353413    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353417    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353421    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353425    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353429    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353436    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353440    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353444    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353448    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353453    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353457    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353461    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353465    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353470    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353474    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353478    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353483    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353487    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.358322 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353491    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353496    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353500    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353504    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353511    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353518    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353524    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353530    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353535    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353540    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353544    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353548    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353553    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353559    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353564    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353569    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353573    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353578    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.358805 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353584    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353589    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353594    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353598    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353602    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353607    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353611    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353615    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353620    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353624    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353628    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353633    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353637    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353641    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353645    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353649    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353654    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353658    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353662    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.353667    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.359275 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353777    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353788    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353798    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353805    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353811    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353817    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353823    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353830    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353836    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353841    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353847    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353852    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353858    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353865    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353872    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353877    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353881    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353886    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353893    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353897    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353902    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353908    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353913    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353919    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:09:17.359748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353924    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353929    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353935    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353940    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353945    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353950    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353955    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353960    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353966    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353971    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353975    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353980    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353985    2573 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353990    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.353998    2573 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354002    2573 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354007    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354012    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354018    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354024    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354029    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354036    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354040    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354066    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354072    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:09:17.360348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354077    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354082    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354087    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354092    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354097    2573 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354103    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354108    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354113    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354118    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354123    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354128    2573 flags.go:64] FLAG: --help="false"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354132    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354138    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354143    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354148    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354153    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354159    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354164    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354168    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354173    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354177    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354183    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354188    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354193    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:09:17.361005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354198    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354203    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354209    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354213    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354220    2573 flags.go:64] FLAG: --lock-file=""
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354225    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354230    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354236    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354244    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354249    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354254    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354260    2573 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354265    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354271    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354276    2573 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354281    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354288    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354293    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354300    2573 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354304    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354309    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354314    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354319    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354324    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354329    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:09:17.361618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354333    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354346    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354351    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354356    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354361    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354366    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354375    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354379    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354384    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354395    2573 flags.go:64] FLAG: --port="10250"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354400    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354408    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-030c1642e3752044e"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354413    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354418    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354423    2573 flags.go:64] FLAG: --register-node="true"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354427    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354434    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354438    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354443    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354449    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354455    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354460    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354465    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354470    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354475    2573 flags.go:64] FLAG: --runonce="false"
Jul 28 17:09:17.362271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354479    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354484    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354489    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354493    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354498    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354503    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354508    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354513    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354518    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354522    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354527    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354531    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354536    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354541    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354546    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354555    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354560    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354566    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354574    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354578    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354583    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354588    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354593    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354598    2573 flags.go:64] FLAG: --v="2"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354610    2573 flags.go:64] FLAG: --version="false"
Jul 28 17:09:17.362875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354617    2573 flags.go:64] FLAG: --vmodule=""
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354624    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.354630    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354834    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354850    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354854    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354857    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354860    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354863    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354867    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354870    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354872    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354876    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354879    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354882    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354885    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354888    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354891    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354893    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354896    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.363517 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354899    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354902    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354908    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354912    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354915    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354918    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354922    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354927    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354931    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354934    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354937    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354940    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354944    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354948    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354952    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354955    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354960    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354963    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354966    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.364007 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354969    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354972    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354975    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354977    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354980    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354982    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354985    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354988    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354990    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354993    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354995    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.354999    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355002    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355005    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355008    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355011    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355014    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355017    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355020    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355023    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.364474 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355025    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355028    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355030    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355033    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355035    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355038    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355040    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355043    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355056    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355060    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355063    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355066    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355069    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355072    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355075    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355077    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355080    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355082    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355085    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.364999 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355087    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355090    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355092    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355095    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355097    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355100    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355102    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355105    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355107    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355110    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355113    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355115    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355118    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355120    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355123    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355125    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355128    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355130    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355133    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355135    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.365481 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355138    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355140    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355146    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.355149    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.355156    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.363666    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.363680    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.363698    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.363704    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363775    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363780    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363784    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363787    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363790    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363793    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.365971 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363795    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363798    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363800    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363803    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363806    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363809    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363811    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363814    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363817    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363819    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363822    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363825    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363835    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363837    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363840    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363842    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363845    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363847    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363850    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363853    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.366348 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363856    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363858    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363861    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363864    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363868    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363872    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363875    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363878    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363881    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363885    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363888    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363891    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363894    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363897    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363899    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363902    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363904    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363907    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363910    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.366825 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363912    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363915    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363918    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363920    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363923    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363926    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363934    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363936    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363939    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363942    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363944    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363947    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363950    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363952    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363955    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363958    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363960    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363963    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363966    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363968    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.367350 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363971    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363974    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363976    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363978    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363981    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363984    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363986    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363990    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363993    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363996    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.363998    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364001    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364003    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364006    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364008    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364011    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364014    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364016    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364019    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.367828 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364028    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364030    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364033    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364035    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364038    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364040    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364043    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364061    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364065    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364069    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364073    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364075    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364078    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364081    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364084    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.368325 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.364089    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364241    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364247    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364251    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364254    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364257    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364260    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364263    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364266    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364268    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364271    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364274    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364277    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364279    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364282    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364284    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364287    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364290    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364298    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:17.368721 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364301    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364303    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364306    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364308    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364311    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364314    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364317    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364319    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364321    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364324    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364326    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364329    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364331    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364334    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364336    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364339    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364341    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364344    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364346    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364349    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:17.369198 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364351    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364354    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364356    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364359    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364361    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364363    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364366    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364368    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364371    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364379    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364382    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364384    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364393    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364396    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364398    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364400    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364403    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364406    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364408    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364411    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:17.369672 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364413    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364415    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364418    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364420    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364423    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364425    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364427    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364430    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364432    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364435    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364438    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364441    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364443    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364445    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364448    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364450    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364452    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364455    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364457    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:17.370212 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364460    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364462    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364465    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364468    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364470    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364473    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364481    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364484    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364487    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364491    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364494    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364497    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364500    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364503    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364505    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364508    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364511    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364514    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364516    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:17.370658 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364519    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364521    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:17.364529    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.364534    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.364648    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.369571    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.370793    2573 server.go:1019] "Starting client certificate rotation"
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.370893    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:17.371161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.370936    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:17.373128 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.373110    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:17.376689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.376650    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:17.385883 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.385864    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:09:17.387654 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.387638    2573 log.go:25] "Validated CRI v1 image API"
Jul 28 17:09:17.390651 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.390631    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:09:17.393695 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.393677    2573 fs.go:135] Filesystem UUIDs: map[4c6acc85-8f58-46c6-a895-ebc661dac7ae:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cf13240a-6e85-462b-a840-6cee71bd3789:/dev/nvme0n1p4]
Jul 28 17:09:17.393753 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.393696    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:09:17.398700 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.398576    2573 manager.go:216] Machine: {Timestamp:2026-07-28 17:09:17.397305291 +0000 UTC m=+0.325226938 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101871 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa600e2849e05e2091f7585aaf05a SystemUUID:ec2fa600-e284-9e05-e209-1f7585aaf05a BootID:f989befe-c784-48bb-8f8c-2f63c8ed0465 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2d:07:dd:e2:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2d:07:dd:e2:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:c6:c3:8f:1f:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:09:17.398700 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.398686    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:09:17.398829 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.398760    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:09:17.401392 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.401368    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:09:17.401525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.401395    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:09:17.402480 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.402470    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:09:17.402512 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.402483    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:09:17.402512 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.402490    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:17.403224 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.403214    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:17.404030 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.404020    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:09:17.405851 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.405840    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:17.405966 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.405957    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:09:17.405996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.405987    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:09:17.406026 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406003    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:09:17.406072 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406029    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:09:17.406072 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406043    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:09:17.406072 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406067    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:09:17.406277 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406261    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:17.406496 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406484    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:17.406553 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.406503    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:17.408526 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.408509    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:09:17.410207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.410194    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:09:17.410252 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.410224    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:09:17.411520 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411506    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411523    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411529    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411535    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411543    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411548    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411558    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411566    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:09:17.411578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411579    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:09:17.411794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411585    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:09:17.411794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411597    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:09:17.411794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.411603    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:09:17.412334 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.412324    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:09:17.412365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.412335    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:09:17.415535 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.415521    2573 server.go:1268] "Started kubelet"
Jul 28 17:09:17.415650 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.415624    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:09:17.415865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.415605    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:09:17.415865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.415827    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:09:17.416357 ip-10-0-129-82 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:09:17.419658 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.416787    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:09:17.419964 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.419939    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:09:17.421500 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.421482    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:09:17.422214 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.422172    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:09:17.422291 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.422222    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:09:17.422651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.422632    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:09:17.424900 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.422093    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-82.ec2.internal.18c6833302fa1611  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-82.ec2.internal,UID:ip-10-0-129-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-82.ec2.internal,},FirstTimestamp:2026-07-28 17:09:17.415495185 +0000 UTC m=+0.343416832,LastTimestamp:2026-07-28 17:09:17.415495185 +0000 UTC m=+0.343416832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-82.ec2.internal,}"
Jul 28 17:09:17.427565 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.427547    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:09:17.427647 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.427575    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:17.427698 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.427666    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428158    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428236    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428254    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428374    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428385    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428428    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428441    2573 factory.go:55] Registering systemd factory
Jul 28 17:09:17.428598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428451    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:09:17.428991 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428685    2573 factory.go:153] Registering CRI-O factory
Jul 28 17:09:17.428991 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428698    2573 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:09:17.428991 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428715    2573 factory.go:103] Registering Raw factory
Jul 28 17:09:17.428991 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.428728    2573 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:09:17.429249 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.429233    2573 manager.go:349] Starting recovery of all containers
Jul 28 17:09:17.430440 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.430403    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:09:17.430575 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.430557    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:09:17.439231 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.439040    2573 manager.go:354] Recovery completed
Jul 28 17:09:17.440461 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.440443    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 17:09:17.443476 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.443463    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.445673 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.445651    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.445730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.445686    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.445730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.445697    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.446151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.446135    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:09:17.446151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.446150    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:09:17.446287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.446153    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.446287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.446170    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:17.447681 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.447666    2573 policy_none.go:49] "None policy: Start"
Jul 28 17:09:17.447757 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.447684    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:09:17.447757 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.447644    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-82.ec2.internal.18c6833304c6907a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-82.ec2.internal,UID:ip-10-0-129-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-82.ec2.internal,},FirstTimestamp:2026-07-28 17:09:17.445673082 +0000 UTC m=+0.373594729,LastTimestamp:2026-07-28 17:09:17.445673082 +0000 UTC m=+0.373594729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-82.ec2.internal,}"
Jul 28 17:09:17.447757 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.447698    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:09:17.448086 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.448041    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.448180 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.448094    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.448180 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.448108    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.448858 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.448845    2573 policy_none.go:47] "Start"
Jul 28 17:09:17.448924 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.448910    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k9mlb"
Jul 28 17:09:17.458083 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.457994    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-82.ec2.internal.18c6833304c6d4fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-82.ec2.internal,UID:ip-10-0-129-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-82.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-82.ec2.internal,},FirstTimestamp:2026-07-28 17:09:17.44569062 +0000 UTC m=+0.373612270,LastTimestamp:2026-07-28 17:09:17.44569062 +0000 UTC m=+0.373612270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-82.ec2.internal,}"
Jul 28 17:09:17.458517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.458499    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k9mlb"
Jul 28 17:09:17.490596 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.490577    2573 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:09:17.490710 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.490611    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:09:17.490710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.490621    2573 server.go:87] "Starting device plugin registration server"
Jul 28 17:09:17.490883 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.490870    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:09:17.490942 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.490885    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:09:17.490995 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.490984    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:09:17.491091 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.491078    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:09:17.491091 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.491091    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:09:17.491633 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.491610    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:09:17.491720 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.491656    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.591301 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.591265    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.592502 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.592471    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.592502 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.592500    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.592606 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.592511    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.592606 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.592542    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.594352 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.594330    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:09:17.595567 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.595547    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:09:17.595696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.595581    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:09:17.595696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.595602    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:09:17.595696 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.595633    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:09:17.597377 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.597356    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:17.598649 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.598636    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.598720 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.598655    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-82.ec2.internal\": node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.646627 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.646595    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.696224 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.696174    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"]
Jul 28 17:09:17.696335 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.696287    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.697853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.697836    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.697944 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.697872    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.697944 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.697887    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.699255 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.699237    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.699376 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.699357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.699421 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.699395    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.699986 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.699971    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.700080 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.700003    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.700080 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.700017    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.700080 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.699973    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.700080 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.700078    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.700267 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.700088    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.701234 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.701211    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.701341 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.701239    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:17.701963 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.701946    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:17.702064 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.701977    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:17.702064 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.701992    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:17.726737 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.726713    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-82.ec2.internal\" not found" node="ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.730099 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.730080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.730197 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.730112    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.730197 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.730137    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7134973092cf17d72d638832558ef84a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-82.ec2.internal\" (UID: \"7134973092cf17d72d638832558ef84a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.731163 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.731149    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-82.ec2.internal\" not found" node="ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.747543 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.747527    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.830645 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.830793 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.830793 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7134973092cf17d72d638832558ef84a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-82.ec2.internal\" (UID: \"7134973092cf17d72d638832558ef84a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.830793 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.830793 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2709edcd733a0c2d129337c27fb71270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal\" (UID: \"2709edcd733a0c2d129337c27fb71270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.830793 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:17.830753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7134973092cf17d72d638832558ef84a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-82.ec2.internal\" (UID: \"7134973092cf17d72d638832558ef84a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:17.848168 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.848097    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:17.949397 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:17.949355    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:18.034057 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.034021    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:18.034130 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.034069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:18.050108 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.050086    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:18.151320 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.151217    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:18.252371 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.252338    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:18.353475 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.353440    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-82.ec2.internal\" not found"
Jul 28 17:09:18.366685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.366661    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:18.370644 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.370629    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:09:18.370752 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.370728    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:09:18.370804 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.370772    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:09:18.407205 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.407138    2573 apiserver.go:52] "Watching apiserver"
Jul 28 17:09:18.427759 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.427732    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:18.427759 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.427753    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:18.430426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.430407    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:09:18.432038 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.432019    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b6rlw","openshift-cluster-node-tuning-operator/tuned-glh7s","openshift-dns/node-resolver-cs2zt","openshift-ovn-kubernetes/ovnkube-node-nlq27","kube-system/global-pull-secret-syncer-b6j9f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf","openshift-image-registry/node-ca-bs5m9","openshift-multus/multus-additional-cni-plugins-l6brt","openshift-multus/multus-m82sq","openshift-multus/network-metrics-daemon-lc55j","openshift-network-diagnostics/network-check-target-t4wgv","openshift-network-operator/iptables-alerter-dht2j"]
Jul 28 17:09:18.433787 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.433773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.434909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.434891    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.436103 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.436084    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.436752 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.436728    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:09:18.436840 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.436808    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:09:18.436840 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.436839    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zlxrn\""
Jul 28 17:09:18.438109 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.437679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.438109 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.438010    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.438727 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.438704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwbgf\""
Jul 28 17:09:18.438809 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.438787    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.439981 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.439935    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.440108 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.440093    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.440215 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.440187    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vl2z5\""
Jul 28 17:09:18.440360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.440334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.440822 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.440770    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:18.440919 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.440883    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:09:18.441088 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.441069    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d2869\""
Jul 28 17:09:18.441252 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.441236    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.441544 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.441529    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.442017 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.442001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.442733 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.442716    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:09:18.442807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.442719    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:09:18.442850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.442820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:09:18.443375 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.443351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.444797 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.444777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.445217 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.445194    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.445312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.445243    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.445312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.445265    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:09:18.445473 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.445320    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bm2jz\""
Jul 28 17:09:18.446045 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.446031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.446795 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.446779    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.446867 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.446841    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:09:18.446933 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.446843    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wksfn\""
Jul 28 17:09:18.446933 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.446843    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.447545 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.447528    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:18.447620 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.447604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.447714 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.447692    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:18.448115 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:09:18.448221 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448120    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:09:18.448221 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448103    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:09:18.448445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448429    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.448529 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448434    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xr86f\""
Jul 28 17:09:18.448529 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.448968 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.448952    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:09:18.449200 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.449184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:18.449260 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.449245    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:18.449837 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.449820    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k4c84\""
Jul 28 17:09:18.450025 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.450007    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:18.450131 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.450095    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"
Jul 28 17:09:18.450510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.450491    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.450953 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.450935    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal"]
Jul 28 17:09:18.454725 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.454705    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jjrll\""
Jul 28 17:09:18.454725 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.454717    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:09:18.454867 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.454738    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:18.455627 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.455612    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:18.462679 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.462638    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:04:17 +0000 UTC" deadline="2028-03-12 01:36:09.595533187 +0000 UTC"
Jul 28 17:09:18.462736 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.462681    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14216h26m51.132855991s"
Jul 28 17:09:18.464182 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.464166    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal"]
Jul 28 17:09:18.464334 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.464323    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:18.476431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.476415    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzbdn"
Jul 28 17:09:18.485978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.485960    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzbdn"
Jul 28 17:09:18.528976 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.528957    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:09:18.537189 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-etc-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537290 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndwb\" (UniqueName: \"kubernetes.io/projected/c681d353-73da-48e0-9cbc-adfe172ed5f7-kube-api-access-bndwb\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537290 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-os-release\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537290 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-hostroot\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-daemon-config\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-env-overrides\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:18.537445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-var-lib-kubelet\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.537445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c55a2e30-9526-43fb-922b-998c5eff59fa-iptables-alerter-script\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-config\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwv9p\" (UniqueName: \"kubernetes.io/projected/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-kube-api-access-dwv9p\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-dbus\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2l2\" (UniqueName: \"kubernetes.io/projected/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-kube-api-access-zn2l2\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-var-lib-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537689 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovn-node-metrics-cert\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537727    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-os-release\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537773    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-socket-dir-parent\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860ffcf-5fdc-47f3-af20-66a41e9bd479-tmp-dir\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-system-cni-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-multus\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-multus-certs\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-kubernetes\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.537987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-sys\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.537997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-host\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3708ca42-b526-496a-9c39-bf295698f0f6-konnectivity-ca\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-systemd\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5x2\" (UniqueName: \"kubernetes.io/projected/c55a2e30-9526-43fb-922b-998c5eff59fa-kube-api-access-qm5x2\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-system-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cni-binary-copy\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-tmp\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8860ffcf-5fdc-47f3-af20-66a41e9bd479-hosts-file\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfccz\" (UniqueName: \"kubernetes.io/projected/8860ffcf-5fdc-47f3-af20-66a41e9bd479-kube-api-access-xfccz\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-script-lib\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-cnibin\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-lib-modules\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538465    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-etc-kubernetes\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.538555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-systemd-units\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-log-socket\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-ovn\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-bin\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cnibin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-k8s-cni-cncf-io\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-conf-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhs7k\" (UniqueName: \"kubernetes.io/projected/4bd7d11a-7276-49b2-8853-48eef0b3b70a-kube-api-access-xhs7k\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55a2e30-9526-43fb-922b-998c5eff59fa-host-slash\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-kubelet\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-slash\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-systemd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-modprobe-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysconfig\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539312 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-netd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghgb\" (UniqueName: \"kubernetes.io/projected/8067b205-077a-4371-a852-08575e036681-kube-api-access-dghgb\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-host\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqn7m\" (UniqueName: \"kubernetes.io/projected/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kube-api-access-nqn7m\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538966    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-netns\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.538981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-run\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-netns\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539112    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-kubelet\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-conf\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-tuned\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-serviceca\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.539865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.540438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539311    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-kubelet-config\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.540438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3708ca42-b526-496a-9c39-bf295698f0f6-agent-certs\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.540438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-node-log\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.540438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-bin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.540438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.539408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhzr\" (UniqueName: \"kubernetes.io/projected/cae1727b-dacc-439b-b01d-81f6b2bdc431-kube-api-access-mjhzr\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.640543 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-kubelet\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640543 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-slash\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-systemd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-modprobe-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysconfig\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-slash\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-kubelet\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-netd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysconfig\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-systemd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-netd\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640720    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-modprobe-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-d\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.640796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghgb\" (UniqueName: \"kubernetes.io/projected/8067b205-077a-4371-a852-08575e036681-kube-api-access-dghgb\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-host\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqn7m\" (UniqueName: \"kubernetes.io/projected/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kube-api-access-nqn7m\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-netns\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-run\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.640957    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-netns\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.640985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-host\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.641048    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.140997946 +0000 UTC m=+2.068919589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641064    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-netns\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-netns\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-run\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.641319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641194    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-kubelet\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-conf\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-tuned\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-serviceca\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-kubelet\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-kubelet-config\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-sysctl-conf\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3708ca42-b526-496a-9c39-bf295698f0f6-agent-certs\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-node-log\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-kubelet-config\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-bin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-bin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhzr\" (UniqueName: \"kubernetes.io/projected/cae1727b-dacc-439b-b01d-81f6b2bdc431-kube-api-access-mjhzr\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.642087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-etc-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndwb\" (UniqueName: \"kubernetes.io/projected/c681d353-73da-48e0-9cbc-adfe172ed5f7-kube-api-access-bndwb\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-os-release\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641618    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-hostroot\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-daemon-config\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-env-overrides\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-etc-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-serviceca\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-var-lib-kubelet\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c55a2e30-9526-43fb-922b-998c5eff59fa-iptables-alerter-script\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-config\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwv9p\" (UniqueName: \"kubernetes.io/projected/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-kube-api-access-dwv9p\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-dbus\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.642972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2l2\" (UniqueName: \"kubernetes.io/projected/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-kube-api-access-zn2l2\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-var-lib-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.641995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovn-node-metrics-cert\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642049    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-var-lib-kubelet\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642197    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-var-lib-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-env-overrides\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.642360    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-node-log\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.642432    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.142413188 +0000 UTC m=+2.070334833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642366    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-daemon-config\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-os-release\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.643863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642429    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-hostroot\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-os-release\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642491    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-socket-dir-parent\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860ffcf-5fdc-47f3-af20-66a41e9bd479-tmp-dir\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de7c57c7-30f7-4d87-ae8a-1c548da27068-dbus\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-system-cni-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-multus\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-multus-certs\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-kubernetes\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-os-release\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642725    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-sys\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-host\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642768    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3708ca42-b526-496a-9c39-bf295698f0f6-konnectivity-ca\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-socket-dir-parent\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.644781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c55a2e30-9526-43fb-922b-998c5eff59fa-iptables-alerter-script\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-system-cni-dir\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642891    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-kubernetes\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-multus-certs\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642956    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-host\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.642753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-var-lib-cni-multus\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-sys\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-systemd\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5x2\" (UniqueName: \"kubernetes.io/projected/c55a2e30-9526-43fb-922b-998c5eff59fa-kube-api-access-qm5x2\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860ffcf-5fdc-47f3-af20-66a41e9bd479-tmp-dir\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-config\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-system-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cni-binary-copy\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-systemd\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-tmp\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8860ffcf-5fdc-47f3-af20-66a41e9bd479-hosts-file\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.645659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643246    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-openvswitch\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfccz\" (UniqueName: \"kubernetes.io/projected/8860ffcf-5fdc-47f3-af20-66a41e9bd479-kube-api-access-xfccz\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-script-lib\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-cnibin\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643375    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3708ca42-b526-496a-9c39-bf295698f0f6-konnectivity-ca\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-lib-modules\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-etc-kubernetes\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643527    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-etc-kubernetes\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-system-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-systemd-units\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643591    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-log-socket\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-lib-modules\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-ovn\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-log-socket\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-bin\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.646525 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cnibin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-k8s-cni-cncf-io\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-conf-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhs7k\" (UniqueName: \"kubernetes.io/projected/4bd7d11a-7276-49b2-8853-48eef0b3b70a-kube-api-access-xhs7k\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55a2e30-9526-43fb-922b-998c5eff59fa-host-slash\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8067b205-077a-4371-a852-08575e036681-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55a2e30-9526-43fb-922b-998c5eff59fa-host-slash\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-systemd-units\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-host-run-k8s-cni-cncf-io\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.644842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-conf-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.645376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cni-binary-copy\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.645529    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-etc-tuned\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.645711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovn-node-metrics-cert\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.643236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.645894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8860ffcf-5fdc-47f3-af20-66a41e9bd479-hosts-file\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8067b205-077a-4371-a852-08575e036681-cnibin\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-multus-cni-dir\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.647406 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646202    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-host-cni-bin\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.648271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd7d11a-7276-49b2-8853-48eef0b3b70a-cnibin\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.648271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c681d353-73da-48e0-9cbc-adfe172ed5f7-run-ovn\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.648271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.646951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c681d353-73da-48e0-9cbc-adfe172ed5f7-ovnkube-script-lib\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.649531 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.649505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-tmp\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.650182 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.650159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3708ca42-b526-496a-9c39-bf295698f0f6-agent-certs\") pod \"konnectivity-agent-b6rlw\" (UID: \"3708ca42-b526-496a-9c39-bf295698f0f6\") " pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.667237 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.667169    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:18.667237 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.667208    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:18.667237 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.667224    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:18.667472 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.667304    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:19.167283557 +0000 UTC m=+2.095205216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:18.668875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.668830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghgb\" (UniqueName: \"kubernetes.io/projected/8067b205-077a-4371-a852-08575e036681-kube-api-access-dghgb\") pod \"multus-additional-cni-plugins-l6brt\" (UID: \"8067b205-077a-4371-a852-08575e036681\") " pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.669228 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.669203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2l2\" (UniqueName: \"kubernetes.io/projected/9605dad1-12b3-4da6-ac7e-a08f7654b6ed-kube-api-access-zn2l2\") pod \"node-ca-bs5m9\" (UID: \"9605dad1-12b3-4da6-ac7e-a08f7654b6ed\") " pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.669372 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.669228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhzr\" (UniqueName: \"kubernetes.io/projected/cae1727b-dacc-439b-b01d-81f6b2bdc431-kube-api-access-mjhzr\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:18.669435 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.669377    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfccz\" (UniqueName: \"kubernetes.io/projected/8860ffcf-5fdc-47f3-af20-66a41e9bd479-kube-api-access-xfccz\") pod \"node-resolver-cs2zt\" (UID: \"8860ffcf-5fdc-47f3-af20-66a41e9bd479\") " pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.669435 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.669412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwv9p\" (UniqueName: \"kubernetes.io/projected/42440cc8-8b92-40b8-afa9-94a5df1f8b7a-kube-api-access-dwv9p\") pod \"tuned-glh7s\" (UID: \"42440cc8-8b92-40b8-afa9-94a5df1f8b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.669885 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.669864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5x2\" (UniqueName: \"kubernetes.io/projected/c55a2e30-9526-43fb-922b-998c5eff59fa-kube-api-access-qm5x2\") pod \"iptables-alerter-dht2j\" (UID: \"c55a2e30-9526-43fb-922b-998c5eff59fa\") " pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.671197 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.671180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqn7m\" (UniqueName: \"kubernetes.io/projected/b9c711b3-67f0-4ee9-9b74-7b3f887d743d-kube-api-access-nqn7m\") pod \"aws-ebs-csi-driver-node-s7xqf\" (UID: \"b9c711b3-67f0-4ee9-9b74-7b3f887d743d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.672106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.672056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhs7k\" (UniqueName: \"kubernetes.io/projected/4bd7d11a-7276-49b2-8853-48eef0b3b70a-kube-api-access-xhs7k\") pod \"multus-m82sq\" (UID: \"4bd7d11a-7276-49b2-8853-48eef0b3b70a\") " pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.673771 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.673748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndwb\" (UniqueName: \"kubernetes.io/projected/c681d353-73da-48e0-9cbc-adfe172ed5f7-kube-api-access-bndwb\") pod \"ovnkube-node-nlq27\" (UID: \"c681d353-73da-48e0-9cbc-adfe172ed5f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.715692 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.715652    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:18.726562 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.726526    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2709edcd733a0c2d129337c27fb71270.slice/crio-c6f4d3b2edd141feb47bde78170c0da12818cc8c07142f4a7159e7f431e7dda8 WatchSource:0}: Error finding container c6f4d3b2edd141feb47bde78170c0da12818cc8c07142f4a7159e7f431e7dda8: Status 404 returned error can't find the container with id c6f4d3b2edd141feb47bde78170c0da12818cc8c07142f4a7159e7f431e7dda8
Jul 28 17:09:18.727026 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.727006    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7134973092cf17d72d638832558ef84a.slice/crio-7508198248a2df0cee773770a4995a3668f327cb7931fd3b3a404433ccd1c05a WatchSource:0}: Error finding container 7508198248a2df0cee773770a4995a3668f327cb7931fd3b3a404433ccd1c05a: Status 404 returned error can't find the container with id 7508198248a2df0cee773770a4995a3668f327cb7931fd3b3a404433ccd1c05a
Jul 28 17:09:18.730983 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.730967    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:09:18.769240 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.769204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:18.775095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.775078    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glh7s"
Jul 28 17:09:18.777565 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.777536    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3708ca42_b526_496a_9c39_bf295698f0f6.slice/crio-bf626d1f0b5732ad0b442dfb8a9be138b31a58e3b59afde57b626788b71fd79d WatchSource:0}: Error finding container bf626d1f0b5732ad0b442dfb8a9be138b31a58e3b59afde57b626788b71fd79d: Status 404 returned error can't find the container with id bf626d1f0b5732ad0b442dfb8a9be138b31a58e3b59afde57b626788b71fd79d
Jul 28 17:09:18.783478 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.783454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42440cc8_8b92_40b8_afa9_94a5df1f8b7a.slice/crio-95777cdfbd34ad2b5f266b2dd78717b79f862cb57ed62ce82016123c68d266e0 WatchSource:0}: Error finding container 95777cdfbd34ad2b5f266b2dd78717b79f862cb57ed62ce82016123c68d266e0: Status 404 returned error can't find the container with id 95777cdfbd34ad2b5f266b2dd78717b79f862cb57ed62ce82016123c68d266e0
Jul 28 17:09:18.792923 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.792903    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cs2zt"
Jul 28 17:09:18.795582 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.795562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:18.801846 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.801780    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8860ffcf_5fdc_47f3_af20_66a41e9bd479.slice/crio-e057ff1b9243478c4804f51062e613a2054f80848fe19a78d527028666277212 WatchSource:0}: Error finding container e057ff1b9243478c4804f51062e613a2054f80848fe19a78d527028666277212: Status 404 returned error can't find the container with id e057ff1b9243478c4804f51062e613a2054f80848fe19a78d527028666277212
Jul 28 17:09:18.804988 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.804957    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc681d353_73da_48e0_9cbc_adfe172ed5f7.slice/crio-bd034228d16fbdb41fbf05dbeefd39ccf655777e9ce07b102bf2afa6220f0a42 WatchSource:0}: Error finding container bd034228d16fbdb41fbf05dbeefd39ccf655777e9ce07b102bf2afa6220f0a42: Status 404 returned error can't find the container with id bd034228d16fbdb41fbf05dbeefd39ccf655777e9ce07b102bf2afa6220f0a42
Jul 28 17:09:18.828874 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.828849    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf"
Jul 28 17:09:18.833303 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.833276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs5m9"
Jul 28 17:09:18.837011 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.836989    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c711b3_67f0_4ee9_9b74_7b3f887d743d.slice/crio-5fbfa699d741aa9e11d7bca56efa3e065c61a3acc7cd932b3512ae2b79b290f5 WatchSource:0}: Error finding container 5fbfa699d741aa9e11d7bca56efa3e065c61a3acc7cd932b3512ae2b79b290f5: Status 404 returned error can't find the container with id 5fbfa699d741aa9e11d7bca56efa3e065c61a3acc7cd932b3512ae2b79b290f5
Jul 28 17:09:18.841882 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.841861    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9605dad1_12b3_4da6_ac7e_a08f7654b6ed.slice/crio-6a4353b4c9b890181773c92094a1d272b4b4be85c9839ba346321375948fa5e6 WatchSource:0}: Error finding container 6a4353b4c9b890181773c92094a1d272b4b4be85c9839ba346321375948fa5e6: Status 404 returned error can't find the container with id 6a4353b4c9b890181773c92094a1d272b4b4be85c9839ba346321375948fa5e6
Jul 28 17:09:18.848556 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.848538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6brt"
Jul 28 17:09:18.856309 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.856283    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8067b205_077a_4371_a852_08575e036681.slice/crio-4ac65b28bb5c2cec240e8e6bacfd120d087eae6f6bab178e1e5f20f60033b868 WatchSource:0}: Error finding container 4ac65b28bb5c2cec240e8e6bacfd120d087eae6f6bab178e1e5f20f60033b868: Status 404 returned error can't find the container with id 4ac65b28bb5c2cec240e8e6bacfd120d087eae6f6bab178e1e5f20f60033b868
Jul 28 17:09:18.866215 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.866196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m82sq"
Jul 28 17:09:18.872389 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:18.872366    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dht2j"
Jul 28 17:09:18.873668 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.873643    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd7d11a_7276_49b2_8853_48eef0b3b70a.slice/crio-96ed3e41a97a469483bde47638342ec07671c446f376f34abf409ca61ee809a2 WatchSource:0}: Error finding container 96ed3e41a97a469483bde47638342ec07671c446f376f34abf409ca61ee809a2: Status 404 returned error can't find the container with id 96ed3e41a97a469483bde47638342ec07671c446f376f34abf409ca61ee809a2
Jul 28 17:09:18.880168 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:18.880148    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55a2e30_9526_43fb_922b_998c5eff59fa.slice/crio-ec61b97e8c4a2711f7bdd638c9498f0a56e5282e47e2c097f40be7b176af35a8 WatchSource:0}: Error finding container ec61b97e8c4a2711f7bdd638c9498f0a56e5282e47e2c097f40be7b176af35a8: Status 404 returned error can't find the container with id ec61b97e8c4a2711f7bdd638c9498f0a56e5282e47e2c097f40be7b176af35a8
Jul 28 17:09:18.881885 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.881867    2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-dht2j_openshift-network-operator(c55a2e30-9526-43fb-922b-998c5eff59fa): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 28 17:09:18.881934 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:18.881899    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-dht2j" podUID="c55a2e30-9526-43fb-922b-998c5eff59fa"
Jul 28 17:09:19.030698 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.030605    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:19.148832 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.148793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:19.149012 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.148877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:19.149012 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.148964    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:19.149012 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.148985    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:19.149178 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.149038    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.149022187 +0000 UTC m=+3.076943822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:19.149178 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.149055    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.149046597 +0000 UTC m=+3.076968231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:19.249485 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.249443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:19.249673 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.249650    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:19.249744 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.249683    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:19.249744 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.249697    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:19.249914 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.249771    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.249750056 +0000 UTC m=+3.177671707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:19.488330 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.488079    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:18 +0000 UTC" deadline="2027-12-27 16:12:08.705961748 +0000 UTC"
Jul 28 17:09:19.488330 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.488147    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12407h2m49.217819654s"
Jul 28 17:09:19.591336 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.591302    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:19.599344 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.599316    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:19.599506 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.599424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:19.630613 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.627982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf" event={"ID":"b9c711b3-67f0-4ee9-9b74-7b3f887d743d","Type":"ContainerStarted","Data":"5fbfa699d741aa9e11d7bca56efa3e065c61a3acc7cd932b3512ae2b79b290f5"}
Jul 28 17:09:19.647566 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.647432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs2zt" event={"ID":"8860ffcf-5fdc-47f3-af20-66a41e9bd479","Type":"ContainerStarted","Data":"e057ff1b9243478c4804f51062e613a2054f80848fe19a78d527028666277212"}
Jul 28 17:09:19.668902 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.668850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b6rlw" event={"ID":"3708ca42-b526-496a-9c39-bf295698f0f6","Type":"ContainerStarted","Data":"bf626d1f0b5732ad0b442dfb8a9be138b31a58e3b59afde57b626788b71fd79d"}
Jul 28 17:09:19.683701 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.683659    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal" event={"ID":"2709edcd733a0c2d129337c27fb71270","Type":"ContainerStarted","Data":"c6f4d3b2edd141feb47bde78170c0da12818cc8c07142f4a7159e7f431e7dda8"}
Jul 28 17:09:19.700716 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.700675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal" event={"ID":"7134973092cf17d72d638832558ef84a","Type":"ContainerStarted","Data":"7508198248a2df0cee773770a4995a3668f327cb7931fd3b3a404433ccd1c05a"}
Jul 28 17:09:19.712737 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.712054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dht2j" event={"ID":"c55a2e30-9526-43fb-922b-998c5eff59fa","Type":"ContainerStarted","Data":"ec61b97e8c4a2711f7bdd638c9498f0a56e5282e47e2c097f40be7b176af35a8"}
Jul 28 17:09:19.718860 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:19.718826    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-dht2j" podUID="c55a2e30-9526-43fb-922b-998c5eff59fa"
Jul 28 17:09:19.724457 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.724423    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m82sq" event={"ID":"4bd7d11a-7276-49b2-8853-48eef0b3b70a","Type":"ContainerStarted","Data":"96ed3e41a97a469483bde47638342ec07671c446f376f34abf409ca61ee809a2"}
Jul 28 17:09:19.740326 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.740006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerStarted","Data":"4ac65b28bb5c2cec240e8e6bacfd120d087eae6f6bab178e1e5f20f60033b868"}
Jul 28 17:09:19.755032 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.754995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs5m9" event={"ID":"9605dad1-12b3-4da6-ac7e-a08f7654b6ed","Type":"ContainerStarted","Data":"6a4353b4c9b890181773c92094a1d272b4b4be85c9839ba346321375948fa5e6"}
Jul 28 17:09:19.769781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.769744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"bd034228d16fbdb41fbf05dbeefd39ccf655777e9ce07b102bf2afa6220f0a42"}
Jul 28 17:09:19.787674 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:19.787640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glh7s" event={"ID":"42440cc8-8b92-40b8-afa9-94a5df1f8b7a","Type":"ContainerStarted","Data":"95777cdfbd34ad2b5f266b2dd78717b79f862cb57ed62ce82016123c68d266e0"}
Jul 28 17:09:20.156613 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.156476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:20.156613 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.156554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:20.156914 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.156629    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:20.156914 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.156679    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:20.156914 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.156709    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.156685385 +0000 UTC m=+5.084607037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:20.156914 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.156734    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.156717962 +0000 UTC m=+5.084639599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:20.257651 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.257614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:20.257862 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.257840    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:20.257942 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.257869    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:20.257942 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.257883    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:20.258057 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.257947    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.257926074 +0000 UTC m=+5.185847732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:20.488896 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.488767    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:18 +0000 UTC" deadline="2028-03-19 02:52:24.384812881 +0000 UTC"
Jul 28 17:09:20.488896 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.488832    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14385h43m3.895985521s"
Jul 28 17:09:20.595933 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.595894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:20.596184 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.596037    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:20.596563 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:20.596538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:20.596682 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:20.596660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:21.032744 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:21.032708    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:21.604044 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:21.604012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:21.604564 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:21.604170    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:22.175338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:22.175416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.175545    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.175613    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.175592525 +0000 UTC m=+9.103514164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.176060    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.176217 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.176119    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.176102997 +0000 UTC m=+9.104024639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.277254 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:22.276448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:22.277254 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.276663    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:22.277254 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.276689    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:22.277254 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.276703    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.277254 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.276768    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.276749115 +0000 UTC m=+9.204670754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.596862 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:22.596012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:22.596862 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.596178    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:22.596862 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:22.596603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:22.596862 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:22.596714    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:23.596883 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:23.596845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:23.597418 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:23.597000    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:24.596087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:24.596030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:24.596087 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:24.596084    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:24.596340 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:24.596165    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:24.596340 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:24.596292    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:25.596465 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:25.595987    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:25.596465 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:25.596149    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:26.209037 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:26.208988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:26.209212 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:26.209087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:26.209212 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.209164    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:26.209212 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.209197    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:26.209381 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.209248    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:34.209227253 +0000 UTC m=+17.137148903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:26.209381 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.209271    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:34.209260798 +0000 UTC m=+17.137182446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:26.309791 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:26.309756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:26.309951 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.309936    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:26.310005 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.309959    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:26.310005 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.309972    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:26.310155 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.310032    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:34.310013394 +0000 UTC m=+17.237935033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:26.596379 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:26.596294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:26.596542 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.596426    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:26.596542 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:26.596474    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:26.596948 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:26.596608    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:27.599531 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:27.599493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:27.599967 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:27.599617    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:28.596426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:28.596383    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:28.596610 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:28.596387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:28.596610 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:28.596520    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:28.596736 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:28.596631    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:29.596252 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:29.596218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:29.596713 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:29.596362    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:30.595798 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:30.595761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:30.595905 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:30.595770    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:30.595905 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:30.595890    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:30.596000 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:30.595959    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:31.596334 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:31.596058    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:31.596749 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:31.596415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:32.596297 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:32.596253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:32.596479 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:32.596269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:32.596479 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:32.596378    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:32.596479 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:32.596446    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:33.596769 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:33.596739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:33.597230 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:33.596868    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:34.267595 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:34.267568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:34.267728 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:34.267608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:34.267728 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.267688    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:34.267728 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.267691    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:34.267728 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.267729    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:50.267717377 +0000 UTC m=+33.195639012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:34.267862 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.267742    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:50.267735704 +0000 UTC m=+33.195657338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:34.368774 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:34.368744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:34.368921 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.368873    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:34.368921 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.368889    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:34.368921 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.368898    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:34.369069 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.368941    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:50.368929352 +0000 UTC m=+33.296850987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:34.596259 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:34.596144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:34.596259 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:34.596195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:34.596487 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.596268    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:34.596487 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:34.596356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:35.596446 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:35.596410    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:35.596978 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:35.596552    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:36.596150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:36.596120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:36.596337 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:36.596237    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:36.596337 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:36.596249    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:36.596455 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:36.596353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:37.596381 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:37.596351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:37.596860 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:37.596432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:38.596158 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:38.596129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:38.596303 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:38.596133    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:38.596303 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:38.596224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:38.596383 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:38.596317    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:39.596746 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.596582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:39.597303 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:39.596824    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:39.839573 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.839537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cs2zt" event={"ID":"8860ffcf-5fdc-47f3-af20-66a41e9bd479","Type":"ContainerStarted","Data":"47ea9544ac421105faf6df7ede95dd27112f320c5f7a50740d7c63327b30c3a8"}
Jul 28 17:09:39.840941 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.840918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b6rlw" event={"ID":"3708ca42-b526-496a-9c39-bf295698f0f6","Type":"ContainerStarted","Data":"bf77bdf8ef500cd68442de788c4b4e45141db86182a3a3fb05e9bf1bc9213501"}
Jul 28 17:09:39.842326 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.842305    2573 generic.go:356] "Generic (PLEG): container finished" podID="2709edcd733a0c2d129337c27fb71270" containerID="dfd31d6617b9ade7a2ab1d1ee95fb8eea7451d9b84a41fa12e844f09357b437d" exitCode=0
Jul 28 17:09:39.842413 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.842365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal" event={"ID":"2709edcd733a0c2d129337c27fb71270","Type":"ContainerDied","Data":"dfd31d6617b9ade7a2ab1d1ee95fb8eea7451d9b84a41fa12e844f09357b437d"}
Jul 28 17:09:39.843783 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.843756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal" event={"ID":"7134973092cf17d72d638832558ef84a","Type":"ContainerStarted","Data":"d1e48ec7c28187aed87807a650791b129fb2a7dff9b8f5842ffad697a8ea4486"}
Jul 28 17:09:39.845337 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.845271    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m82sq" event={"ID":"4bd7d11a-7276-49b2-8853-48eef0b3b70a","Type":"ContainerStarted","Data":"2ae2f89aa8f127bdad8af1527a1e8fd1b766f52c6279ca54e0a37e448feaf036"}
Jul 28 17:09:39.846699 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.846677    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="8fb4f47ba4bb10caf6fcd54ef7aad63bb3406c6bf50662195dd7a52931266e10" exitCode=0
Jul 28 17:09:39.846790 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.846751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"8fb4f47ba4bb10caf6fcd54ef7aad63bb3406c6bf50662195dd7a52931266e10"}
Jul 28 17:09:39.850683 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.850657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs5m9" event={"ID":"9605dad1-12b3-4da6-ac7e-a08f7654b6ed","Type":"ContainerStarted","Data":"0ccb78f78cac160861ed7aa7f74d79852c85c347491dadde03a5b3b3863c1fd2"}
Jul 28 17:09:39.853477 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"482753df4ecc7dbdaac1114bb4ec6e388d2fcb5234f5fad306b2ee9625d93e68"}
Jul 28 17:09:39.853567 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"54000012e8c0db140993924f2538ce2630abb0a33bca4bdad98309af42d8949f"}
Jul 28 17:09:39.853629 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"65e08647b3d787e0cb147e14e1e06bced32a1ef779be30e6a9b1f59f4e1798d8"}
Jul 28 17:09:39.853682 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"3950e99c042f07c1e84271c94030b9578da71cfabd341829e3d963f7d9ac55ea"}
Jul 28 17:09:39.853750 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853699    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"00d3da8dd19d070e7f5813a45ccbae2e109cdfa8d08082aa4bcb466e8d18d8a8"}
Jul 28 17:09:39.853807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.853747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"1df30f1a058c7c87bb3f569aa2020880eebbb76629f34fe1867703d5e2c5fe88"}
Jul 28 17:09:39.854775 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.854754    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glh7s" event={"ID":"42440cc8-8b92-40b8-afa9-94a5df1f8b7a","Type":"ContainerStarted","Data":"69d2635ba16aa41c816beda9b0fa7e058da97f312a3791fa834e9ea5d67f782c"}
Jul 28 17:09:39.855962 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.855947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf" event={"ID":"b9c711b3-67f0-4ee9-9b74-7b3f887d743d","Type":"ContainerStarted","Data":"61a18e7dbd32f47cce9cf0ae01737c38bcc6a49bab9fd2106ea8fe3d1ffd6b0e"}
Jul 28 17:09:39.863805 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.863758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cs2zt" podStartSLOduration=2.018164343 podStartE2EDuration="21.863746948s" totalImagesPullingTime="19.845582605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:39.863671086 +0000 UTC m=+22.791592743" watchObservedRunningTime="2026-07-28 17:09:39.863746948 +0000 UTC m=+22.791668605"
Jul 28 17:09:39.903033 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.902995    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b6rlw" podStartSLOduration=6.510795453 podStartE2EDuration="21.902982256s" totalImagesPullingTime="15.392186803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:39.902812285 +0000 UTC m=+22.830733941" watchObservedRunningTime="2026-07-28 17:09:39.902982256 +0000 UTC m=+22.830903909"
Jul 28 17:09:39.921000 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.920967    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-glh7s" podStartSLOduration=2.054355157 podStartE2EDuration="21.920954364s" totalImagesPullingTime="19.866599207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:39.920939229 +0000 UTC m=+22.848860886" watchObservedRunningTime="2026-07-28 17:09:39.920954364 +0000 UTC m=+22.848875998"
Jul 28 17:09:39.943489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.943402    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bs5m9" podStartSLOduration=2.137370129 podStartE2EDuration="21.943388826s" totalImagesPullingTime="19.806018697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:39.942839184 +0000 UTC m=+22.870760840" watchObservedRunningTime="2026-07-28 17:09:39.943388826 +0000 UTC m=+22.871310481"
Jul 28 17:09:39.991746 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:39.991699    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m82sq" podStartSLOduration=2.204903908 podStartE2EDuration="21.991684697s" totalImagesPullingTime="19.786780789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:39.991514466 +0000 UTC m=+22.919436118" watchObservedRunningTime="2026-07-28 17:09:39.991684697 +0000 UTC m=+22.919606354"
Jul 28 17:09:40.011604 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.011567    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.014533 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.014501    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-82.ec2.internal" podStartSLOduration=22.014488113 podStartE2EDuration="22.014488113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.013948815 +0000 UTC m=+22.941870471" watchObservedRunningTime="2026-07-28 17:09:40.014488113 +0000 UTC m=+22.942409769"
Jul 28 17:09:40.110286 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.110253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.195655 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.195589    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.252171 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.252134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.596263 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.596193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:40.596263 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.596223    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:40.596633 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:40.596356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:40.596633 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:40.596530    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:40.861291 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.861199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal" event={"ID":"2709edcd733a0c2d129337c27fb71270","Type":"ContainerStarted","Data":"e4978930108f335878a4032aeba1d17df2a65c27417ab32a956cb070bd9af83d"}
Jul 28 17:09:40.863894 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.863765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dht2j" event={"ID":"c55a2e30-9526-43fb-922b-998c5eff59fa","Type":"ContainerStarted","Data":"7c7c0909412057690afa86c926ba72da692edd4977994df4e66f66234565f1e8"}
Jul 28 17:09:40.879644 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.879593    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-82.ec2.internal" podStartSLOduration=22.879578476 podStartE2EDuration="22.879578476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.878636851 +0000 UTC m=+23.806558509" watchObservedRunningTime="2026-07-28 17:09:40.879578476 +0000 UTC m=+23.807500132"
Jul 28 17:09:40.895118 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.895072    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dht2j" podStartSLOduration=22.895043789 podStartE2EDuration="22.895043789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 17:09:40.894924118 +0000 UTC m=+23.822845775" watchObservedRunningTime="2026-07-28 17:09:40.895043789 +0000 UTC m=+23.822965447"
Jul 28 17:09:40.944764 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:40.944734    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:09:41.018796 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.018764    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:41.084191 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.084155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:41.503070 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.502931    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:40.944758915Z","UUID":"d797b2d8-d33c-4183-833f-64eb9b1f358d","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:09:41.504764 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.504739    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:09:41.505020 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.504784    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:09:41.596355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.596326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:41.596536 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:41.596457    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:41.869235 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:41.869203    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf" event={"ID":"b9c711b3-67f0-4ee9-9b74-7b3f887d743d","Type":"ContainerStarted","Data":"1a674e7348ce3866960779e929e5bda2bbee67bd0af9059f93d841ec7b804047"}
Jul 28 17:09:42.596129 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:42.596039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:42.596281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:42.596057    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:42.596281 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:42.596163    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:42.596389 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:42.596288    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:42.875178 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:42.875091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"cf409370e7455bcf3ca62069d6576ec606a3648062a4407d505d8224e0a33bac"}
Jul 28 17:09:42.877416 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:42.877383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf" event={"ID":"b9c711b3-67f0-4ee9-9b74-7b3f887d743d","Type":"ContainerStarted","Data":"5919407717a5c3c8f78c65e1ecb04bd3fc21928497d9382f54d8a7bfcbdad701"}
Jul 28 17:09:42.899167 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:42.899112    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xqf" podStartSLOduration=2.3205430910000002 podStartE2EDuration="24.899095584s" totalImagesPullingTime="22.578552493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:42.898641131 +0000 UTC m=+25.826562789" watchObservedRunningTime="2026-07-28 17:09:42.899095584 +0000 UTC m=+25.827017240"
Jul 28 17:09:43.071411 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.071365    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:43.139113 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.139012    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:43.596658 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.596583    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:43.596805 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:43.596706    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:43.810088 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.810037    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:43.810645 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.810625    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:43.879921 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.879831    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:43.880348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.880197    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b6rlw"
Jul 28 17:09:43.949732 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:43.949679    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:44.596937 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.596692    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:44.597117 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.596693    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:44.597117 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:44.596977    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:44.597117 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:44.597036    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:44.883911 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.883824    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="4991dae6ed2b08ca8d950d500e1ecf0fd8ffbe5c4dac5296fb52c5b1e1d9a20d" exitCode=0
Jul 28 17:09:44.883911 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.883860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"4991dae6ed2b08ca8d950d500e1ecf0fd8ffbe5c4dac5296fb52c5b1e1d9a20d"}
Jul 28 17:09:44.887678 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.887652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" event={"ID":"c681d353-73da-48e0-9cbc-adfe172ed5f7","Type":"ContainerStarted","Data":"72a93933b298a69d942392ba5136d84860306d3862701f9c3ba5a53c8ae1a72d"}
Jul 28 17:09:44.887902 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.887883    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:44.902643 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.902624    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:44.943119 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:44.943075    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27" podStartSLOduration=6.903718018 podStartE2EDuration="26.943037737s" totalImagesPullingTime="20.039319719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:44.9411719 +0000 UTC m=+27.869093555" watchObservedRunningTime="2026-07-28 17:09:44.943037737 +0000 UTC m=+27.870959393"
Jul 28 17:09:45.596035 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:45.595991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:45.596233 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:45.596169    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:45.891300 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:45.890972    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:45.891300 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:45.891297    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:45.905724 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:45.905695    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:09:46.366436 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.366402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6j9f"]
Jul 28 17:09:46.366589 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.366512    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:46.366650 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:46.366597    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:46.370117 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.370086    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t4wgv"]
Jul 28 17:09:46.370651 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.370201    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:46.370651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:46.370282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:46.370774 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.370733    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc55j"]
Jul 28 17:09:46.370839 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.370823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:46.370924 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:46.370907    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:46.894760 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.894727    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="33798809c58620a6c5e7ba4ac4b7137c96f4b99869dc30b034ae7f6ecabd8e6c" exitCode=0
Jul 28 17:09:46.895152 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:46.894817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"33798809c58620a6c5e7ba4ac4b7137c96f4b99869dc30b034ae7f6ecabd8e6c"}
Jul 28 17:09:47.899961 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:47.899878    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="c83bedb1c44194cdc0d3a78534f0671e21310c6792869cd01b9454264791dc2c" exitCode=0
Jul 28 17:09:47.900345 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:47.899958    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"c83bedb1c44194cdc0d3a78534f0671e21310c6792869cd01b9454264791dc2c"}
Jul 28 17:09:48.596146 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:48.596063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:48.596146 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:48.596106    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:48.596353 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:48.596177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:48.596353 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:48.596196    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:48.596353 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:48.596272    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:48.596509 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:48.596348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:50.293299 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.293268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:50.293651 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.293320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:50.293651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.293417    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:50.293651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.293418    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:50.293651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.293475    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.293457161 +0000 UTC m=+65.221378795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:50.293651 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.293488    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret podName:de7c57c7-30f7-4d87-ae8a-1c548da27068 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.293482627 +0000 UTC m=+65.221404260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret") pod "global-pull-secret-syncer-b6j9f" (UID: "de7c57c7-30f7-4d87-ae8a-1c548da27068") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:50.394132 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.394096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:50.394299 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.394282    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:50.394358 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.394307    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:50.394358 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.394320    2573 projected.go:196] Error preparing data for projected volume kube-api-access-nt6b8 for pod openshift-network-diagnostics/network-check-target-t4wgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:50.394467 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.394391    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8 podName:17889c5b-ad0a-48cc-aa55-ddc8d356f448 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.394371354 +0000 UTC m=+65.322293008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nt6b8" (UniqueName: "kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8") pod "network-check-target-t4wgv" (UID: "17889c5b-ad0a-48cc-aa55-ddc8d356f448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:50.596475 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.596389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:50.596639 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.596396    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:50.596639 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:50.596517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:50.596639 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.596510    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t4wgv" podUID="17889c5b-ad0a-48cc-aa55-ddc8d356f448"
Jul 28 17:09:50.596639 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.596585    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:09:50.596639 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:50.596628    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6j9f" podUID="de7c57c7-30f7-4d87-ae8a-1c548da27068"
Jul 28 17:09:51.415422 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.415246    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-82.ec2.internal" event="NodeReady"
Jul 28 17:09:51.415875 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.415510    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:09:51.476899 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.476873    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bswg7"]
Jul 28 17:09:51.479861 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.479839    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lkktt"]
Jul 28 17:09:51.479983 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.479968    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.483043 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.483026    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.483301 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.483280    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zj6nk\""
Jul 28 17:09:51.483515 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.483501    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:09:51.483658 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.483640    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:09:51.485939 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.485917    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:09:51.486065 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.485999    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:09:51.486167 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.486151    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:09:51.486265 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.486151    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6sdzh\""
Jul 28 17:09:51.490427 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.490407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lkktt"]
Jul 28 17:09:51.497768 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.497747    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bswg7"]
Jul 28 17:09:51.603783 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.603783 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mh9b\" (UniqueName: \"kubernetes.io/projected/706624fd-df7d-4366-994a-c1a4d310a483-kube-api-access-7mh9b\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.603980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9a6911-897c-43e0-81f0-56627ef9c2f6-cert\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.603980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhcr\" (UniqueName: \"kubernetes.io/projected/8b9a6911-897c-43e0-81f0-56627ef9c2f6-kube-api-access-7qhcr\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.603980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706624fd-df7d-4366-994a-c1a4d310a483-tmp-dir\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.603980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.603961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/706624fd-df7d-4366-994a-c1a4d310a483-config-volume\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.612143 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.612095    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:51.672388 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.672350    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:51.705308 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mh9b\" (UniqueName: \"kubernetes.io/projected/706624fd-df7d-4366-994a-c1a4d310a483-kube-api-access-7mh9b\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.705479 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705342    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9a6911-897c-43e0-81f0-56627ef9c2f6-cert\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.705479 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhcr\" (UniqueName: \"kubernetes.io/projected/8b9a6911-897c-43e0-81f0-56627ef9c2f6-kube-api-access-7qhcr\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.705479 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706624fd-df7d-4366-994a-c1a4d310a483-tmp-dir\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.705479 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/706624fd-df7d-4366-994a-c1a4d310a483-config-volume\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.705678 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.705507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.705678 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:51.705614    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:51.705678 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:51.705669    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:52.205650708 +0000 UTC m=+35.133572343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:51.706041 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.706023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706624fd-df7d-4366-994a-c1a4d310a483-tmp-dir\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.706252 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.706212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/706624fd-df7d-4366-994a-c1a4d310a483-config-volume\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.710114 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.710089    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9a6911-897c-43e0-81f0-56627ef9c2f6-cert\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.715023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.715003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhcr\" (UniqueName: \"kubernetes.io/projected/8b9a6911-897c-43e0-81f0-56627ef9c2f6-kube-api-access-7qhcr\") pod \"ingress-canary-lkktt\" (UID: \"8b9a6911-897c-43e0-81f0-56627ef9c2f6\") " pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:51.715536 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.715507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mh9b\" (UniqueName: \"kubernetes.io/projected/706624fd-df7d-4366-994a-c1a4d310a483-kube-api-access-7mh9b\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:51.795765 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:51.795725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lkktt"
Jul 28 17:09:52.209549 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.209514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:52.209717 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:52.209702    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:52.209784 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:52.209773    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:53.209755638 +0000 UTC m=+36.137677274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:52.595889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.595804    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:09:52.596350 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.595806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:09:52.596350 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.595806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:09:52.601296 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601268    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:09:52.601464 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601310    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-24g9l\""
Jul 28 17:09:52.601464 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601334    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:09:52.601464 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601351    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-65cz5\""
Jul 28 17:09:52.601653 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601628    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:09:52.601764 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:52.601683    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:09:53.220310 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:53.220278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:53.220455 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:53.220438    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:53.220515 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:53.220504    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:55.220488577 +0000 UTC m=+38.148410210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:53.274537 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:53.274493    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lkktt"]
Jul 28 17:09:53.325736 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:09:53.325696    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9a6911_897c_43e0_81f0_56627ef9c2f6.slice/crio-46c9b9cdea843a6f1b7f4ad954d05ea8d416fe95ef7a9938a21dff9a070cdefa WatchSource:0}: Error finding container 46c9b9cdea843a6f1b7f4ad954d05ea8d416fe95ef7a9938a21dff9a070cdefa: Status 404 returned error can't find the container with id 46c9b9cdea843a6f1b7f4ad954d05ea8d416fe95ef7a9938a21dff9a070cdefa
Jul 28 17:09:53.917672 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:53.917476    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="73c4d2eebb6cd20e59a179476f17c2724b809d49854ae989c2c63cbe3e071738" exitCode=0
Jul 28 17:09:53.918538 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:53.917570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"73c4d2eebb6cd20e59a179476f17c2724b809d49854ae989c2c63cbe3e071738"}
Jul 28 17:09:53.919035 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:53.919010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lkktt" event={"ID":"8b9a6911-897c-43e0-81f0-56627ef9c2f6","Type":"ContainerStarted","Data":"46c9b9cdea843a6f1b7f4ad954d05ea8d416fe95ef7a9938a21dff9a070cdefa"}
Jul 28 17:09:54.365809 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:54.365701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.435382 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:54.435266    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.924762 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:54.924733    2573 generic.go:356] "Generic (PLEG): container finished" podID="8067b205-077a-4371-a852-08575e036681" containerID="d61c9a6b7e8e5778a3c149971464b8529ca36637ae071ae16b14107daa14b3e5" exitCode=0
Jul 28 17:09:54.925134 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:54.924793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerDied","Data":"d61c9a6b7e8e5778a3c149971464b8529ca36637ae071ae16b14107daa14b3e5"}
Jul 28 17:09:55.233237 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:55.233162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:55.233380 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:55.233349    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:55.233444 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:55.233430    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.233409679 +0000 UTC m=+42.161331319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:55.930820 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:55.930798    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6brt" event={"ID":"8067b205-077a-4371-a852-08575e036681","Type":"ContainerStarted","Data":"bd8831143727318cb0ee6415b56e05bcc293deca64e39fb51fa3f2ed78abdd74"}
Jul 28 17:09:55.962600 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:55.960364    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l6brt" podStartSLOduration=6.761733908 podStartE2EDuration="37.960342828s" totalImagesPullingTime="31.19860892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:55.957021104 +0000 UTC m=+38.884942761" watchObservedRunningTime="2026-07-28 17:09:55.960342828 +0000 UTC m=+38.888264487"
Jul 28 17:09:56.082279 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.082251    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.152175 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.152067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.693856 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.693816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.792260 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.792232    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.934213 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.934177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lkktt" event={"ID":"8b9a6911-897c-43e0-81f0-56627ef9c2f6","Type":"ContainerStarted","Data":"7da7d321c81bc97d09c068e72cee1481dacc5a1a8231652f30a181f189784093"}
Jul 28 17:09:56.954279 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:56.954175    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lkktt" podStartSLOduration=3.389908046 podStartE2EDuration="5.954156873s" totalImagesPullingTime="2.564248827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:56.953812498 +0000 UTC m=+39.881734150" watchObservedRunningTime="2026-07-28 17:09:56.954156873 +0000 UTC m=+39.882078530"
Jul 28 17:09:59.259926 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:09:59.259886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:09:59.260324 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:59.260062    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:59.260324 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:09:59.260132    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:07.260116374 +0000 UTC m=+50.188038010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:07.315203 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:07.315161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:10:07.315595 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:07.315313    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:07.315595 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:07.315377    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:23.315358798 +0000 UTC m=+66.243280438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:17.917547 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:17.917523    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlq27"
Jul 28 17:10:18.058570 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:18.058528    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.144501 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:18.144463    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.214532 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:18.214447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:22.320174 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.320123    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:10:22.320644 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.320213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:10:22.323470 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.323447    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:22.323577 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.323450    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:22.331017 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:22.330999    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:10:22.331091 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:22.331081    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:26.331040998 +0000 UTC m=+129.258962639 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : secret "metrics-daemon-secret" not found
Jul 28 17:10:22.334426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.334410    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de7c57c7-30f7-4d87-ae8a-1c548da27068-original-pull-secret\") pod \"global-pull-secret-syncer-b6j9f\" (UID: \"de7c57c7-30f7-4d87-ae8a-1c548da27068\") " pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:10:22.420955 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.420916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:10:22.424460 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.424443    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:10:22.434939 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.434919    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:10:22.445482 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.445456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6b8\" (UniqueName: \"kubernetes.io/projected/17889c5b-ad0a-48cc-aa55-ddc8d356f448-kube-api-access-nt6b8\") pod \"network-check-target-t4wgv\" (UID: \"17889c5b-ad0a-48cc-aa55-ddc8d356f448\") " pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:10:22.610944 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.610860    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-65cz5\""
Jul 28 17:10:22.617751 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.617731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:10:22.620030 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.620012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6j9f"
Jul 28 17:10:22.753468 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.753437    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t4wgv"]
Jul 28 17:10:22.756410 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:10:22.756379    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17889c5b_ad0a_48cc_aa55_ddc8d356f448.slice/crio-51e50d019968027354e5d6a5967bd608bbe2604a58e1326df6c2a120ec246d06 WatchSource:0}: Error finding container 51e50d019968027354e5d6a5967bd608bbe2604a58e1326df6c2a120ec246d06: Status 404 returned error can't find the container with id 51e50d019968027354e5d6a5967bd608bbe2604a58e1326df6c2a120ec246d06
Jul 28 17:10:22.769481 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.768405    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6j9f"]
Jul 28 17:10:22.772543 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:10:22.772520    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7c57c7_30f7_4d87_ae8a_1c548da27068.slice/crio-98d0998029d1f12725bbd6a14b7d5ebdef3f59969078d68dcd8bce16a508d506 WatchSource:0}: Error finding container 98d0998029d1f12725bbd6a14b7d5ebdef3f59969078d68dcd8bce16a508d506: Status 404 returned error can't find the container with id 98d0998029d1f12725bbd6a14b7d5ebdef3f59969078d68dcd8bce16a508d506
Jul 28 17:10:22.993314 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.993282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6j9f" event={"ID":"de7c57c7-30f7-4d87-ae8a-1c548da27068","Type":"ContainerStarted","Data":"98d0998029d1f12725bbd6a14b7d5ebdef3f59969078d68dcd8bce16a508d506"}
Jul 28 17:10:22.994378 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:22.994355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t4wgv" event={"ID":"17889c5b-ad0a-48cc-aa55-ddc8d356f448","Type":"ContainerStarted","Data":"51e50d019968027354e5d6a5967bd608bbe2604a58e1326df6c2a120ec246d06"}
Jul 28 17:10:23.327981 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:23.327864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:10:23.328473 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:23.328030    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:23.328473 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:23.328094    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:55.328080583 +0000 UTC m=+98.256002217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:28.008033 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:28.007996    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6j9f" event={"ID":"de7c57c7-30f7-4d87-ae8a-1c548da27068","Type":"ContainerStarted","Data":"90aabcfccf908b60fb07ec1504172ead7380e138d8bda907f4a0a341301c64f3"}
Jul 28 17:10:28.009383 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:28.009355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t4wgv" event={"ID":"17889c5b-ad0a-48cc-aa55-ddc8d356f448","Type":"ContainerStarted","Data":"4e0d520b14bd45464661575ca5131a3c4646f0dc6f97c1a42b086a35584e17fd"}
Jul 28 17:10:28.009513 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:28.009498    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:10:28.028975 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:28.028924    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b6j9f" podStartSLOduration=65.43926935 podStartE2EDuration="1m10.028910712s" totalImagesPullingTime="4.589641362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:28.028292878 +0000 UTC m=+70.956214535" watchObservedRunningTime="2026-07-28 17:10:28.028910712 +0000 UTC m=+70.956832367"
Jul 28 17:10:28.046410 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:28.046361    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t4wgv" podStartSLOduration=65.443783223 podStartE2EDuration="1m10.046345385s" totalImagesPullingTime="4.602562162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:28.045983171 +0000 UTC m=+70.973904829" watchObservedRunningTime="2026-07-28 17:10:28.046345385 +0000 UTC m=+70.974267055"
Jul 28 17:10:45.338697 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:45.338566    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:45.438355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:45.438322    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:55.342349 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:55.342312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:10:55.342870 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:55.342421    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:55.342870 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:10:55.342484    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.342470214 +0000 UTC m=+162.270391848 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:59.014572 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.014538    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t4wgv"
Jul 28 17:10:59.322299 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.322216    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"]
Jul 28 17:10:59.353788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.353751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"]
Jul 28 17:10:59.353953 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.353857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.357414 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.357388    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:10:59.357621 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.357601    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:10:59.359253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.359104    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:10:59.359253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.359122    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:10:59.359253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.359109    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:10:59.359253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.359176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:10:59.359253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.359239    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:10:59.470288 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.470288 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.470528 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.470528 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7gq\" (UniqueName: \"kubernetes.io/projected/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-kube-api-access-wp7gq\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.470528 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.470528 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.470477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571539 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7gq\" (UniqueName: \"kubernetes.io/projected/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-kube-api-access-wp7gq\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.571710 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.571668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.572391 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.572331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.574209 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.574185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-ca\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.574318 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.574244    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.574376 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.574336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.574376 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.574336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-hub\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.582854 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.582829    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7gq\" (UniqueName: \"kubernetes.io/projected/ee3fcca8-d7e2-4b03-a47b-2dc8035fc366-kube-api-access-wp7gq\") pod \"cluster-proxy-proxy-agent-6674df7f-wcr72\" (UID: \"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.673126 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.673093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:10:59.818305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:10:59.818271    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"]
Jul 28 17:10:59.821572 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:10:59.821545    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee3fcca8_d7e2_4b03_a47b_2dc8035fc366.slice/crio-14d427078636ef8022485c86b7f6e44d2832b41a076dbd1749279f7d4434a616 WatchSource:0}: Error finding container 14d427078636ef8022485c86b7f6e44d2832b41a076dbd1749279f7d4434a616: Status 404 returned error can't find the container with id 14d427078636ef8022485c86b7f6e44d2832b41a076dbd1749279f7d4434a616
Jul 28 17:11:00.084111 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:00.084025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerStarted","Data":"14d427078636ef8022485c86b7f6e44d2832b41a076dbd1749279f7d4434a616"}
Jul 28 17:11:04.099065 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:04.099011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerStarted","Data":"49242da9ce835a970dee18fa674f8a8b187103afea2fe01c5e38a24ad8bd3916"}
Jul 28 17:11:04.099065 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:04.099070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerStarted","Data":"f5e1ecdc3ab483fad85e14311493884f47948735f782f2a41f99e61fd1364801"}
Jul 28 17:11:04.099517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:04.099087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerStarted","Data":"7483346cce479c81d65209e44594d640f77343db0b4f03de7d9820ca48d07c05"}
Jul 28 17:11:04.123414 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:04.123364    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" podStartSLOduration=1.47789132 podStartE2EDuration="5.123350072s" totalImagesPullingTime="3.645458752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.121346242 +0000 UTC m=+107.049267899" watchObservedRunningTime="2026-07-28 17:11:04.123350072 +0000 UTC m=+107.051271728"
Jul 28 17:11:26.348778 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:26.348737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:11:26.349260 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:26.348891    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:11:26.349260 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:26.348959    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.34894136 +0000 UTC m=+251.276862993 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : secret "metrics-daemon-secret" not found
Jul 28 17:11:54.491345 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:54.491298    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bswg7" podUID="706624fd-df7d-4366-994a-c1a4d310a483"
Jul 28 17:11:55.240039 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:55.240004    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:11:55.613951 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:55.613867    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:11:59.374120 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:11:59.374080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:11:59.374531 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:59.374207    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:59.374531 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:11:59.374283    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls podName:706624fd-df7d-4366-994a-c1a4d310a483 nodeName:}" failed. No retries permitted until 2026-07-28 17:14:01.374262962 +0000 UTC m=+284.302184597 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls") pod "dns-default-bswg7" (UID: "706624fd-df7d-4366-994a-c1a4d310a483") : secret "dns-default-metrics-tls" not found
Jul 28 17:12:07.598124 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:12:07.598088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:12:47.394928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:12:47.394836    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:12.984946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.984914    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:13:12.986814 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.986798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:12.990169 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.990145    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:13:12.990642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.990626    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:13:12.990849 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.990837    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2dfz6\""
Jul 28 17:13:12.991179 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.991167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:13:12.997107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:12.997087    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:13:13.003696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.003677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:13:13.069619 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069947 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069947 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.069947 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.069868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5bd\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.136513 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.136487    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:13.170620 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5bd\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.170767    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.170783    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567b68ddbf-dc2fg: secret "image-registry-tls" not found
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.170807    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.170903 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.170840    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls podName:31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d nodeName:}" failed. No retries permitted until 2026-07-28 17:13:13.670819531 +0000 UTC m=+236.598741188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls") pod "image-registry-567b68ddbf-dc2fg" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d") : secret "image-registry-tls" not found
Jul 28 17:13:13.171813 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.171796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.171975 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.171951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.172504 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.172480    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.173027 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.173013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.173258 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.173241    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.180484 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.180464    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5bd\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.181198 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.181181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.235617 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.235497    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:13.673425 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:13.673396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:13.673576 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.673540    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:13:13.673576 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.673555    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567b68ddbf-dc2fg: secret "image-registry-tls" not found
Jul 28 17:13:13.673643 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:13.673625    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls podName:31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d nodeName:}" failed. No retries permitted until 2026-07-28 17:13:14.673607673 +0000 UTC m=+237.601529316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls") pod "image-registry-567b68ddbf-dc2fg" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d") : secret "image-registry-tls" not found
Jul 28 17:13:14.680942 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:14.680906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:14.681343 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:14.681061    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:13:14.681343 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:14.681079    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567b68ddbf-dc2fg: secret "image-registry-tls" not found
Jul 28 17:13:14.681343 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:14.681142    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls podName:31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d nodeName:}" failed. No retries permitted until 2026-07-28 17:13:16.681127779 +0000 UTC m=+239.609049431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls") pod "image-registry-567b68ddbf-dc2fg" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d") : secret "image-registry-tls" not found
Jul 28 17:13:16.695641 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:16.695605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:16.696015 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:16.695743    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:13:16.696015 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:16.695761    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567b68ddbf-dc2fg: secret "image-registry-tls" not found
Jul 28 17:13:16.696015 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:16.695824    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls podName:31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d nodeName:}" failed. No retries permitted until 2026-07-28 17:13:20.69580888 +0000 UTC m=+243.623730513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls") pod "image-registry-567b68ddbf-dc2fg" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d") : secret "image-registry-tls" not found
Jul 28 17:13:17.393277 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:17.393246    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:17.459037 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:17.458981    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:17.933240 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:17.933209    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:18.141692 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.141660    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7trjn"]
Jul 28 17:13:18.144276 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.144259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.148410 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.148390    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:13:18.148541 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.148394    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:13:18.148637 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.148614    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:13:18.149884 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.149866    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:13:18.149977 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.149870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hktz8\""
Jul 28 17:13:18.156343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.156322    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7trjn"]
Jul 28 17:13:18.206609 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.206548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-crio-socket\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.206609 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.206575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.206781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.206610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-data-volume\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.206781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.206691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tdt\" (UniqueName: \"kubernetes.io/projected/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-api-access-q5tdt\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.206781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.206756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.307657 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.307466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.307880 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.307860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-crio-socket\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.307980 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.307885    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:18.307980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.307906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.307998    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:18.807972378 +0000 UTC m=+241.735894016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.308039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-data-volume\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.308066    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.308121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tdt\" (UniqueName: \"kubernetes.io/projected/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-api-access-q5tdt\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.308142    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-crio-socket\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.308150    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:18.808132402 +0000 UTC m=+241.736054059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : configmap "kube-rbac-proxy" not found
Jul 28 17:13:18.311315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.308555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-data-volume\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.317012 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.316981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tdt\" (UniqueName: \"kubernetes.io/projected/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-api-access-q5tdt\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.811592 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.811550    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.811866 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:18.811619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:18.811866 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.811699    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:13:18.811866 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.811705    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:18.811866 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.811770    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:19.811754825 +0000 UTC m=+242.739676459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:18.811866 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:18.811788    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:19.811780069 +0000 UTC m=+242.739701703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : configmap "kube-rbac-proxy" not found
Jul 28 17:13:19.216754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:19.216716    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:19.818931 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:19.818890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:19.819120 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:19.818951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:19.819120 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:19.819018    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:13:19.819120 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:19.819069    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:19.819120 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:19.819100    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:21.819085585 +0000 UTC m=+244.747007219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : configmap "kube-rbac-proxy" not found
Jul 28 17:13:19.819260 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:19.819137    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:21.819123606 +0000 UTC m=+244.747045241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:19.900591 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:19.900563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:20.001348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:20.001317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:20.271668 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:20.271627    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:20.335137 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:20.335089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:20.712729 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:20.712704    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs2zt_8860ffcf-5fdc-47f3-af20-66a41e9bd479/dns-node-resolver/0.log"
Jul 28 17:13:20.727096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:20.727043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:20.727208 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:20.727194    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:13:20.727244 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:20.727211    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567b68ddbf-dc2fg: secret "image-registry-tls" not found
Jul 28 17:13:20.727277 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:20.727265    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls podName:31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d nodeName:}" failed. No retries permitted until 2026-07-28 17:13:28.72724608 +0000 UTC m=+251.655167726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls") pod "image-registry-567b68ddbf-dc2fg" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d") : secret "image-registry-tls" not found
Jul 28 17:13:21.221431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:21.221405    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:21.712807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:21.712780    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs5m9_9605dad1-12b3-4da6-ac7e-a08f7654b6ed/node-ca/0.log"
Jul 28 17:13:21.835402 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:21.835361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:21.835564 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:21.835430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:21.835564 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:21.835506    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:13:21.835564 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:21.835562    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:25.83554807 +0000 UTC m=+248.763469704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : configmap "kube-rbac-proxy" not found
Jul 28 17:13:21.835697 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:21.835509    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:21.835697 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:21.835626    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:25.835614694 +0000 UTC m=+248.763536346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:22.512540 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:22.512507    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lkktt_8b9a6911-897c-43e0-81f0-56627ef9c2f6/serve-healthcheck-canary/0.log"
Jul 28 17:13:23.899063 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:23.899022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.004566 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.004528    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.130630 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.130599    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.230007 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.229978    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.333110 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.333067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.431712 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.431667    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.595183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:24.595096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:25.613030 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:25.613000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:25.865325 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:25.865233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:25.865325 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:25.865315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:25.865507 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:25.865355    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:13:25.865507 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:25.865397    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:25.865507 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:25.865420    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:33.86540373 +0000 UTC m=+256.793325364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : configmap "kube-rbac-proxy" not found
Jul 28 17:13:25.865507 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:25.865436    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls podName:a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff nodeName:}" failed. No retries permitted until 2026-07-28 17:13:33.865426019 +0000 UTC m=+256.793347653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7trjn" (UID: "a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:13:26.021070 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:26.021021    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:26.196708 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:26.196673    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:26.390253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:26.390220    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:26.797391 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:26.797357    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:27.405198 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:27.405160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:27.481613 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:27.481486    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:27.812354 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:27.812321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:27.897456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:27.897417    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.320998 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.320966    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.383374 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.383332    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:13:28.383537 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:28.383483    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:13:28.383574 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:28.383549    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs podName:cae1727b-dacc-439b-b01d-81f6b2bdc431 nodeName:}" failed. No retries permitted until 2026-07-28 17:15:30.383533234 +0000 UTC m=+373.311454871 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs") pod "network-metrics-daemon-lc55j" (UID: "cae1727b-dacc-439b-b01d-81f6b2bdc431") : secret "metrics-daemon-secret" not found
Jul 28 17:13:28.607867 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.607785    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.786599 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.786561    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:28.789072 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.789018    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"image-registry-567b68ddbf-dc2fg\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") " pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:28.899958 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.899875    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2dfz6\""
Jul 28 17:13:28.906226 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:28.906208    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:29.013904 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.013874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:29.042281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.042251    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:13:29.045197 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:13:29.045174    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d1b8ba_8c34_4755_b3cd_a5cfd9a4474d.slice/crio-319a281ff75e329fba08a31f29bde5f997813ce5c26913e483c7fefd74a50010 WatchSource:0}: Error finding container 319a281ff75e329fba08a31f29bde5f997813ce5c26913e483c7fefd74a50010: Status 404 returned error can't find the container with id 319a281ff75e329fba08a31f29bde5f997813ce5c26913e483c7fefd74a50010
Jul 28 17:13:29.196906 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.196869    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:29.493794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.493715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" event={"ID":"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d","Type":"ContainerStarted","Data":"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"}
Jul 28 17:13:29.493794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.493761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" event={"ID":"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d","Type":"ContainerStarted","Data":"319a281ff75e329fba08a31f29bde5f997813ce5c26913e483c7fefd74a50010"}
Jul 28 17:13:29.493794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.493791    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:13:29.520743 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.520690    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" podStartSLOduration=17.520673065 podStartE2EDuration="17.520673065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:29.51838005 +0000 UTC m=+252.446301707" watchObservedRunningTime="2026-07-28 17:13:29.520673065 +0000 UTC m=+252.448594721"
Jul 28 17:13:29.598197 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:29.598164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:30.390586 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:30.390552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:30.596612 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:30.596583    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:30.866343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:30.866313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:31.021555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:31.021519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:31.179218 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:31.179187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:31.240193 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:31.240160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:31.599122 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:31.599017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.191720 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:32.191683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.600578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:32.600494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.806108 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:32.806074    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.990526 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:32.990491    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.476066 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.476013    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.925487 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.925457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:33.925849 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.925506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:33.926092 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.926073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:33.927743 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.927723    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7trjn\" (UID: \"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff\") " pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:33.990571 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:33.990538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:34.052472 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.052441    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7trjn"
Jul 28 17:13:34.180565 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.180363    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7trjn"]
Jul 28 17:13:34.183230 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:13:34.183194    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ebd6cb_c7a2_4b1f_b3d9_291980fe1bff.slice/crio-ddba55a5fe7335c3f54d07ed0bd231d590ab01464935f6b9cb6a1402bbe9ec37 WatchSource:0}: Error finding container ddba55a5fe7335c3f54d07ed0bd231d590ab01464935f6b9cb6a1402bbe9ec37: Status 404 returned error can't find the container with id ddba55a5fe7335c3f54d07ed0bd231d590ab01464935f6b9cb6a1402bbe9ec37
Jul 28 17:13:34.201645 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.201613    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:34.398366 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.398328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:34.508559 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.508470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7trjn" event={"ID":"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff","Type":"ContainerStarted","Data":"ddba55a5fe7335c3f54d07ed0bd231d590ab01464935f6b9cb6a1402bbe9ec37"}
Jul 28 17:13:34.592744 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:34.592704    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:39.377149 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:39.377069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:39.479694 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:39.479668    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:39.527891 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:39.527857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7trjn" event={"ID":"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff","Type":"ContainerStarted","Data":"6a0d5b20938ba53e411951fec43fb835394c6b003081f0dac81c219f5d600bc8"}
Jul 28 17:13:43.545442 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.545400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7trjn" event={"ID":"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff","Type":"ContainerStarted","Data":"c085bd35a002aee5e5596da1db9ffbbc2126248904d4d04d443a749e80a1a601"}
Jul 28 17:13:43.559236 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.559191    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:13:43.604519 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.604486    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8fc4f564-xgb8k"]
Jul 28 17:13:43.606808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.606784    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.622658 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.622630    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8fc4f564-xgb8k"]
Jul 28 17:13:43.653092 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.653020    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:43.705094 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-certificates\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705094 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-trusted-ca\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705340 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-bound-sa-token\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705340 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-tls\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705340 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-image-registry-private-configuration\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhlx\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-kube-api-access-bdhlx\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-ca-trust-extracted\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.705604 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.705487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-installation-pull-secrets\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.754649 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.754599    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:43.806741 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.806659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-bound-sa-token\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.806741 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.806698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-tls\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.806741 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.806730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-image-registry-private-configuration\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807014 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.806761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhlx\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-kube-api-access-bdhlx\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807014 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.806793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-ca-trust-extracted\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807167 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.807099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-installation-pull-secrets\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807167 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.807155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-certificates\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807273 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.807190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-trusted-ca\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807273 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.807220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-ca-trust-extracted\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.807939 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.807918    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-certificates\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.808115 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.808095    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-trusted-ca\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.809985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.809928    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-image-registry-private-configuration\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.809985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.809964    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-registry-tls\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.810209 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.810191    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-installation-pull-secrets\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.821633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.821606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-bound-sa-token\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.821824 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.821805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhlx\" (UniqueName: \"kubernetes.io/projected/2be1ca24-53fb-40c9-bc7f-2b83ff9014c5-kube-api-access-bdhlx\") pod \"image-registry-5d8fc4f564-xgb8k\" (UID: \"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5\") " pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:43.917422 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:43.917392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:44.065919 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:44.065834    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8fc4f564-xgb8k"]
Jul 28 17:13:44.068759 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:13:44.068729    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be1ca24_53fb_40c9_bc7f_2b83ff9014c5.slice/crio-465da45a53ff12fadcb5940ae8c0ca90c52bfb0bd4041e80c90b563aa97933ec WatchSource:0}: Error finding container 465da45a53ff12fadcb5940ae8c0ca90c52bfb0bd4041e80c90b563aa97933ec: Status 404 returned error can't find the container with id 465da45a53ff12fadcb5940ae8c0ca90c52bfb0bd4041e80c90b563aa97933ec
Jul 28 17:13:44.550443 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:44.550400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k" event={"ID":"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5","Type":"ContainerStarted","Data":"69e2636587cb9c491f4b55c94798afe5956216e1a5c826e7a95923fb5463e4ca"}
Jul 28 17:13:44.550878 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:44.550461    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k" event={"ID":"2be1ca24-53fb-40c9-bc7f-2b83ff9014c5","Type":"ContainerStarted","Data":"465da45a53ff12fadcb5940ae8c0ca90c52bfb0bd4041e80c90b563aa97933ec"}
Jul 28 17:13:44.550878 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:44.550507    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:13:44.574754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:44.574704    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k" podStartSLOduration=1.574685772 podStartE2EDuration="1.574685772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:44.573097161 +0000 UTC m=+267.501018817" watchObservedRunningTime="2026-07-28 17:13:44.574685772 +0000 UTC m=+267.502607427"
Jul 28 17:13:46.559602 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:46.559563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7trjn" event={"ID":"a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff","Type":"ContainerStarted","Data":"6a2bf58de5ffa3094c628b60656f24101e99764b9ff02a8fecdc670cddb488f5"}
Jul 28 17:13:46.580820 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:46.580771    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7trjn" podStartSLOduration=16.536690098 podStartE2EDuration="28.580756143s" totalImagesPullingTime="12.044066045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:13:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:46.579072381 +0000 UTC m=+269.506994038" watchObservedRunningTime="2026-07-28 17:13:46.580756143 +0000 UTC m=+269.508677801"
Jul 28 17:13:48.639725 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:48.639692    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:48.744471 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:48.744422    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:48.837421 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:48.837388    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:49.236510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:49.236480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:50.329310 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:50.329276    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:52.852843 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:52.852804    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:52.925867 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:52.925829    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:52.980360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:52.980328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:53.564988 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:53.564955    2573 patch_prober.go:28] interesting pod/image-registry-567b68ddbf-dc2fg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 17:13:53.565154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:53.565003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:13:54.628812 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:54.628777    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.042729 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.042691    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.344501 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.344409    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.432276 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.432242    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4gj76"]
Jul 28 17:13:57.439261 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.439231    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.442397 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.442367    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:13:57.443936 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443908    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:13:57.443936 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443912    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:13:57.443936 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443935    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:13:57.444183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443948    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:13:57.444183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443979    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:13:57.444183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.443935    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zd2gj\""
Jul 28 17:13:57.459154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.459124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.515666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-root\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.515833 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-sys\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.515833 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmzt\" (UniqueName: \"kubernetes.io/projected/150c1b4e-b984-4b42-9b0c-9d04f3641320-kube-api-access-cpmzt\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.515833 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-tls\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.515833 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-metrics-client-ca\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.516095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-textfile\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.516095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-wtmp\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.516095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-accelerators-collector-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.516095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.515972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.557151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.557118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.617262 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-wtmp\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617262 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-accelerators-collector-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-root\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-sys\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmzt\" (UniqueName: \"kubernetes.io/projected/150c1b4e-b984-4b42-9b0c-9d04f3641320-kube-api-access-cpmzt\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-tls\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617382    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-wtmp\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-metrics-client-ca\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-root\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/150c1b4e-b984-4b42-9b0c-9d04f3641320-sys\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-textfile\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617925 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-textfile\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617925 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617846    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-accelerators-collector-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.617925 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.617877    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150c1b4e-b984-4b42-9b0c-9d04f3641320-metrics-client-ca\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.619709 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.619691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-tls\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.619805 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.619719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150c1b4e-b984-4b42-9b0c-9d04f3641320-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.627558 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.627537    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmzt\" (UniqueName: \"kubernetes.io/projected/150c1b4e-b984-4b42-9b0c-9d04f3641320-kube-api-access-cpmzt\") pod \"node-exporter-4gj76\" (UID: \"150c1b4e-b984-4b42-9b0c-9d04f3641320\") " pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.740346 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.740309    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.747392 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.747368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4gj76"
Jul 28 17:13:57.758491 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:13:57.758470    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c1b4e_b984_4b42_9b0c_9d04f3641320.slice/crio-8dd798cd1b52bf639d7c0ee40fe23cbacafa270ff4697d97085e5ef283a1dccc WatchSource:0}: Error finding container 8dd798cd1b52bf639d7c0ee40fe23cbacafa270ff4697d97085e5ef283a1dccc: Status 404 returned error can't find the container with id 8dd798cd1b52bf639d7c0ee40fe23cbacafa270ff4697d97085e5ef283a1dccc
Jul 28 17:13:57.883398 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.883307    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:57.986024 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:57.985974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:58.161100 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.161061    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:58.241235 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:13:58.241193    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bswg7" podUID="706624fd-df7d-4366-994a-c1a4d310a483"
Jul 28 17:13:58.476378 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.476261    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:58.595081 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.595061    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:13:58.595207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.595078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4gj76" event={"ID":"150c1b4e-b984-4b42-9b0c-9d04f3641320","Type":"ContainerStarted","Data":"8dd798cd1b52bf639d7c0ee40fe23cbacafa270ff4697d97085e5ef283a1dccc"}
Jul 28 17:13:58.653106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.653025    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:58.872270 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:58.872183    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:59.600127 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:59.600092    2573 generic.go:356] "Generic (PLEG): container finished" podID="150c1b4e-b984-4b42-9b0c-9d04f3641320" containerID="17f8ef83aeb81b39ad3de78cb4a36850c1134ebb459e1e50f7e533a008ff157a" exitCode=0
Jul 28 17:13:59.600503 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:13:59.600152    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4gj76" event={"ID":"150c1b4e-b984-4b42-9b0c-9d04f3641320","Type":"ContainerDied","Data":"17f8ef83aeb81b39ad3de78cb4a36850c1134ebb459e1e50f7e533a008ff157a"}
Jul 28 17:14:00.446238 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.446202    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:00.549898 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.549862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:00.605473 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.605436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4gj76" event={"ID":"150c1b4e-b984-4b42-9b0c-9d04f3641320","Type":"ContainerStarted","Data":"14845ec37b36877d589339bf8760ed3296e56d0b0a7db417b5785e84de12b2de"}
Jul 28 17:14:00.605853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.605491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4gj76" event={"ID":"150c1b4e-b984-4b42-9b0c-9d04f3641320","Type":"ContainerStarted","Data":"7632d8d2bd68ca5724419aa4de5e0e666ff89153d69c85a83207c82bf4a4c7bb"}
Jul 28 17:14:00.620726 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.620693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:00.635684 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.635637    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4gj76" podStartSLOduration=2.7720421010000003 podStartE2EDuration="3.635623444s" totalImagesPullingTime="798.324065ms" totalInitContainerRuntime="65.257278ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:00.634951692 +0000 UTC m=+283.562873347" watchObservedRunningTime="2026-07-28 17:14:00.635623444 +0000 UTC m=+283.563545103"
Jul 28 17:14:00.776283 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:00.776213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:01.095906 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.095825    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:01.234201 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.234160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:01.338552 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.338520    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:01.450914 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.450884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:14:01.453250 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.453229    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/706624fd-df7d-4366-994a-c1a4d310a483-metrics-tls\") pod \"dns-default-bswg7\" (UID: \"706624fd-df7d-4366-994a-c1a4d310a483\") " pod="openshift-dns/dns-default-bswg7"
Jul 28 17:14:01.599850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.599822    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zj6nk\""
Jul 28 17:14:01.606346 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.606325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:14:01.917558 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:01.915870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bswg7"]
Jul 28 17:14:01.920444 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:14:01.920420    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706624fd_df7d_4366_994a_c1a4d310a483.slice/crio-b6b87a879f7a310fd7762c8ebc41ac61d657feb14a822052ded67c052c697630 WatchSource:0}: Error finding container b6b87a879f7a310fd7762c8ebc41ac61d657feb14a822052ded67c052c697630: Status 404 returned error can't find the container with id b6b87a879f7a310fd7762c8ebc41ac61d657feb14a822052ded67c052c697630
Jul 28 17:14:02.174890 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.174849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:02.338704 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.338671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:02.612838 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.612750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswg7" event={"ID":"706624fd-df7d-4366-994a-c1a4d310a483","Type":"ContainerStarted","Data":"b6b87a879f7a310fd7762c8ebc41ac61d657feb14a822052ded67c052c697630"}
Jul 28 17:14:02.670696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.670663    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55f8cc7554-g746n"]
Jul 28 17:14:02.687848 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.687817    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.692770 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.692741    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:14:02.693125 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.693105    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:14:02.694281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.694256    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:14:02.694494 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.694460    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:14:02.694972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.694787    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:14:02.694972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.694899    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p9rz7\""
Jul 28 17:14:02.695216 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.695196    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55f8cc7554-g746n"]
Jul 28 17:14:02.701093 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.701072    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:14:02.864078 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.863979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbch\" (UniqueName: \"kubernetes.io/projected/08029731-101b-4166-a46c-71bae3782f8c-kube-api-access-rhbch\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864078 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-serving-certs-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864078 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-federate-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-telemeter-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.864360 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.864307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-metrics-client-ca\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.965669 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-telemeter-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.965861 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.965861 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-metrics-client-ca\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.965974 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbch\" (UniqueName: \"kubernetes.io/projected/08029731-101b-4166-a46c-71bae3782f8c-kube-api-access-rhbch\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.965974 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-serving-certs-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.966196 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.965974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-federate-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.966196 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.966004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.966196 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.966099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.966740 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.966689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.966929 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.966904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-metrics-client-ca\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.967509 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.967461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08029731-101b-4166-a46c-71bae3782f8c-serving-certs-ca-bundle\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.968931 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.968908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.969214 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.969193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-federate-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.969310 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.969230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.971738 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.971714    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08029731-101b-4166-a46c-71bae3782f8c-telemeter-client-tls\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:02.979273 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:02.979248    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbch\" (UniqueName: \"kubernetes.io/projected/08029731-101b-4166-a46c-71bae3782f8c-kube-api-access-rhbch\") pod \"telemeter-client-55f8cc7554-g746n\" (UID: \"08029731-101b-4166-a46c-71bae3782f8c\") " pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:03.001164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.001143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n"
Jul 28 17:14:03.181713 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.181678    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55f8cc7554-g746n"]
Jul 28 17:14:03.185598 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:14:03.185571    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08029731_101b_4166_a46c_71bae3782f8c.slice/crio-801808a8051b91034a8f93a444c5a6c18c33d6fe961f83aefe4fbc2682483af9 WatchSource:0}: Error finding container 801808a8051b91034a8f93a444c5a6c18c33d6fe961f83aefe4fbc2682483af9: Status 404 returned error can't find the container with id 801808a8051b91034a8f93a444c5a6c18c33d6fe961f83aefe4fbc2682483af9
Jul 28 17:14:03.564949 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.564914    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:14:03.617324 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.617285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n" event={"ID":"08029731-101b-4166-a46c-71bae3782f8c","Type":"ContainerStarted","Data":"801808a8051b91034a8f93a444c5a6c18c33d6fe961f83aefe4fbc2682483af9"}
Jul 28 17:14:03.850173 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.850090    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:03.922407 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.922372    2573 patch_prober.go:28] interesting pod/image-registry-5d8fc4f564-xgb8k container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 17:14:03.922556 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:03.922426    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k" podUID="2be1ca24-53fb-40c9-bc7f-2b83ff9014c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:14:04.623815 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:04.623777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswg7" event={"ID":"706624fd-df7d-4366-994a-c1a4d310a483","Type":"ContainerStarted","Data":"bd534111da9866fd191baea154029eb285396fbc8add1a43c9f82a221995a10a"}
Jul 28 17:14:04.624298 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:04.623828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswg7" event={"ID":"706624fd-df7d-4366-994a-c1a4d310a483","Type":"ContainerStarted","Data":"67499475a93230fa35c46fadc6bbd74a7161d3eddc5f741377215bbfa51ee32b"}
Jul 28 17:14:04.624298 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:04.623929    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:14:04.646703 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:04.646649    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bswg7" podStartSLOduration=251.320115593 podStartE2EDuration="4m13.646630886s" totalImagesPullingTime="2.326515293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:04.645734626 +0000 UTC m=+287.573656281" watchObservedRunningTime="2026-07-28 17:14:04.646630886 +0000 UTC m=+287.574552545"
Jul 28 17:14:05.558659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:05.558635    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8fc4f564-xgb8k"
Jul 28 17:14:05.786127 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:05.786093    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:05.884944 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:05.884913    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:05.985144 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:05.985103    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:06.092096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:06.091992    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:06.160092 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:06.160033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:06.633841 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:06.633803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n" event={"ID":"08029731-101b-4166-a46c-71bae3782f8c","Type":"ContainerStarted","Data":"1a2a7c49f6ba1162174c27525665d13f4ab2565b3abaaa45dbb8c63e795f6ae2"}
Jul 28 17:14:07.115238 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.115175    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.221341 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.221308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.324544 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.324516    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.423193 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.423164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.583478 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.583374    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.638239 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.638203    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n" event={"ID":"08029731-101b-4166-a46c-71bae3782f8c","Type":"ContainerStarted","Data":"16f88b776358eddb85da86d53e824a628d8d7c3ae279aa3f651aa991c3a5531d"}
Jul 28 17:14:07.638390 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.638266    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n" event={"ID":"08029731-101b-4166-a46c-71bae3782f8c","Type":"ContainerStarted","Data":"21d357417a4242f33b51c2f0f982ddb88827a6766ff3c24d4c7f0d52df0d8673"}
Jul 28 17:14:07.652698 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.652637    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.683932 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.683884    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55f8cc7554-g746n" podStartSLOduration=2.17067852 podStartE2EDuration="5.683869529s" totalImagesPullingTime="3.513191009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:07.683036798 +0000 UTC m=+290.610958455" watchObservedRunningTime="2026-07-28 17:14:07.683869529 +0000 UTC m=+290.611791185"
Jul 28 17:14:07.801265 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.801229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:07.991714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:07.991686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:08.138277 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.138243    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:08.374831 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.374746    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:08.435086 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.435037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:08.582517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.582477    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerName="registry" containerID="cri-o://228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897" gracePeriod=30
Jul 28 17:14:08.824499 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.824478    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:14:08.915333 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915279    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll5bd\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915333 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915328    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915376    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915410    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915463    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915494    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915527    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.915576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915555    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets\") pod \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\" (UID: \"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d\") "
Jul 28 17:14:08.916021 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.915965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:08.916170 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.916032    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:08.918041 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.917976    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:14:08.918164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.918086    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd" (OuterVolumeSpecName: "kube-api-access-ll5bd") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "kube-api-access-ll5bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:14:08.918164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.918106    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:14:08.918164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.918086    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:14:08.918164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.918153    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:14:08.923463 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:08.923442    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" (UID: "31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016869    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-trusted-ca\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016894    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-image-registry-private-configuration\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016903    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-certificates\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016913    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-installation-pull-secrets\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016921    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll5bd\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-kube-api-access-ll5bd\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016929    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-registry-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016937    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-ca-trust-extracted\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.016946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.016945    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d-bound-sa-token\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:14:09.024585 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.024562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:09.240070 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.240016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:09.302824 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.302762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:09.649107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.649077    2573 generic.go:356] "Generic (PLEG): container finished" podID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerID="228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897" exitCode=0
Jul 28 17:14:09.649229 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.649115    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg"
Jul 28 17:14:09.649229 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.649132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" event={"ID":"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d","Type":"ContainerDied","Data":"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"}
Jul 28 17:14:09.649229 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.649174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567b68ddbf-dc2fg" event={"ID":"31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d","Type":"ContainerDied","Data":"319a281ff75e329fba08a31f29bde5f997813ce5c26913e483c7fefd74a50010"}
Jul 28 17:14:09.649229 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.649196    2573 scope.go:117] "RemoveContainer" containerID="228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"
Jul 28 17:14:09.657024 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.657007    2573 scope.go:117] "RemoveContainer" containerID="228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"
Jul 28 17:14:09.657285 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:14:09.657267    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897\": container with ID starting with 228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897 not found: ID does not exist" containerID="228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"
Jul 28 17:14:09.657354 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.657292    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897"} err="failed to get container status \"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897\": rpc error: code = NotFound desc = could not find container \"228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897\": container with ID starting with 228e524cd3861d30139dde623d087e195789f6635f84475138f5ce5216fba897 not found: ID does not exist"
Jul 28 17:14:09.668609 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.668589    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:14:09.672942 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:09.672924    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-567b68ddbf-dc2fg"]
Jul 28 17:14:10.214779 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:10.214744    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:10.325005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:10.324978    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:10.598838 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:14:10.598752    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lc55j" podUID="cae1727b-dacc-439b-b01d-81f6b2bdc431"
Jul 28 17:14:11.600987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:11.600953    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" path="/var/lib/kubelet/pods/31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d/volumes"
Jul 28 17:14:14.077715 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:14.077676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:14.151261 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:14.151227    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:14.629630 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:14.629606    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bswg7"
Jul 28 17:14:17.446914 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.446889    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:14:17.447345 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.447272    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerName="registry"
Jul 28 17:14:17.447345 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.447297    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerName="registry"
Jul 28 17:14:17.447453 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.447359    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d1b8ba-8c34-4755-b3cd-a5cfd9a4474d" containerName="registry"
Jul 28 17:14:17.451131 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.451110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.453027 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.453006    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:14:17.460246 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.460231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:14:17.461463 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.461447    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:14:17.462019 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.462004    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:14:17.462236 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.462212    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:14:17.462335 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.462249    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:14:17.462335 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.462278    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:14:17.462466 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.462337    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:14:17.464346 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.464328    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:14:17.469637 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.469616    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dnxr4\""
Jul 28 17:14:17.472902 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.472882    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.472978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.472917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.472978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.472936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.472978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.472961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjgw\" (UniqueName: \"kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.473101 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.473022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.473101 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.473064    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.533766 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.533730    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:17.573533 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.573533 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.573722 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.573784 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjgw\" (UniqueName: \"kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.573838 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.573872 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.573853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.574283 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.574254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.574283 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.574278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.574469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.574439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.576023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.575990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.576156 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.576099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.585257 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.585236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjgw\" (UniqueName: \"kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw\") pod \"console-6cbc9d5947-qqd54\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") " pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.689869 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.689831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:17.759800 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.759725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:17.910511 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:17.910483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:14:17.913998 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:14:17.913970    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87366516_387b_42e0_a8bd_b529842ef75f.slice/crio-ba83106176f7bb4dcc03154114be2794dfc86a58a71aed1ee8eb9261e36bbda4 WatchSource:0}: Error finding container ba83106176f7bb4dcc03154114be2794dfc86a58a71aed1ee8eb9261e36bbda4: Status 404 returned error can't find the container with id ba83106176f7bb4dcc03154114be2794dfc86a58a71aed1ee8eb9261e36bbda4
Jul 28 17:14:18.681340 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:18.681286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbc9d5947-qqd54" event={"ID":"87366516-387b-42e0-a8bd-b529842ef75f","Type":"ContainerStarted","Data":"ba83106176f7bb4dcc03154114be2794dfc86a58a71aed1ee8eb9261e36bbda4"}
Jul 28 17:14:20.688991 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:20.688957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbc9d5947-qqd54" event={"ID":"87366516-387b-42e0-a8bd-b529842ef75f","Type":"ContainerStarted","Data":"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"}
Jul 28 17:14:20.713460 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:20.713395    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cbc9d5947-qqd54" podStartSLOduration=1.007863359 podStartE2EDuration="3.713378842s" totalImagesPullingTime="2.705515483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:20.711010274 +0000 UTC m=+303.638931930" watchObservedRunningTime="2026-07-28 17:14:20.713378842 +0000 UTC m=+303.641300501"
Jul 28 17:14:20.835106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:20.835069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:20.934166 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:20.934129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:22.454787 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:22.454748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:22.551535 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:22.551490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:23.163385 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:23.163340    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:23.291005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:23.290917    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:23.600403 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:23.600325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:14:23.808027 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:23.807988    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.328373 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:24.328341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.433808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:24.433769    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.020247 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.020218    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:14:26.023420 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.023402    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.033323 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.033301    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:14:26.034983 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.034961    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:14:26.144228 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144606 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvpr\" (UniqueName: \"kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.144606 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.144487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.203437 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.203406    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.245009 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.244988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvpr\" (UniqueName: \"kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245226 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245266 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245266 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.245976 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.246042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.245972    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.246147 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.246130    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.247509 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.247489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.247621 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.247607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.254571 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.254546    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvpr\" (UniqueName: \"kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr\") pod \"console-5c78f98f4c-dwqnv\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") " pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.305065 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.304970    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.331987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.331961    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:26.465869 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.461811    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:14:26.534730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.534703    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/init-textfile/0.log"
Jul 28 17:14:26.708341 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.708302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78f98f4c-dwqnv" event={"ID":"b7be772f-6755-4459-bb18-1da0b2dd3723","Type":"ContainerStarted","Data":"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"}
Jul 28 17:14:26.708555 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.708391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78f98f4c-dwqnv" event={"ID":"b7be772f-6755-4459-bb18-1da0b2dd3723","Type":"ContainerStarted","Data":"913a636e52e2ea1a352555d4d974c73b7b559ba5e597ddb3b8fed033593d12f8"}
Jul 28 17:14:26.732816 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.732768    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c78f98f4c-dwqnv" podStartSLOduration=1.7327489310000002 podStartE2EDuration="1.732748931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:26.732320996 +0000 UTC m=+309.660242652" watchObservedRunningTime="2026-07-28 17:14:26.732748931 +0000 UTC m=+309.660670587"
Jul 28 17:14:26.742311 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.742286    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/node-exporter/0.log"
Jul 28 17:14:26.934393 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:26.934362    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/kube-rbac-proxy/0.log"
Jul 28 17:14:27.760200 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:27.760173    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:27.760200 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:27.760224    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:27.764824 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:27.764805    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:28.718272 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:28.718244    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:14:30.936146 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:30.936112    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/telemeter-client/0.log"
Jul 28 17:14:31.135069 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:31.134986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/reload/0.log"
Jul 28 17:14:31.334085 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:31.333990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/kube-rbac-proxy/0.log"
Jul 28 17:14:33.336625 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:33.336595    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cbc9d5947-qqd54_87366516-387b-42e0-a8bd-b529842ef75f/console/0.log"
Jul 28 17:14:34.137140 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:34.137112    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bswg7_706624fd-df7d-4366-994a-c1a4d310a483/dns/0.log"
Jul 28 17:14:34.337218 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:34.337187    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bswg7_706624fd-df7d-4366-994a-c1a4d310a483/kube-rbac-proxy/0.log"
Jul 28 17:14:34.935941 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:34.935918    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs2zt_8860ffcf-5fdc-47f3-af20-66a41e9bd479/dns-node-resolver/0.log"
Jul 28 17:14:36.332140 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:36.332110    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:36.332556 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:36.332169    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:36.337136 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:36.337112    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:36.749172 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:36.749139    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:14:36.802074 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:36.800246    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:14:37.205071 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:37.205014    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:37.305655 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:37.305623    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:37.871223 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:37.871115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:37.968615 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:37.968579    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:38.421758 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:38.421722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:38.522485 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:38.522457    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:39.334912 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:39.334875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:39.406985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:39.406949    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:49.675144 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:49.675100    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" podUID="ee3fcca8-d7e2-4b03-a47b-2dc8035fc366" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:14:54.330715 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:54.330682    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:54.428615 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:54.428578    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:57.807533 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:57.807494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:57.926355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:57.926322    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:59.674938 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:14:59.674904    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" podUID="ee3fcca8-d7e2-4b03-a47b-2dc8035fc366" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:15:01.831149 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:01.831076    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cbc9d5947-qqd54" podUID="87366516-387b-42e0-a8bd-b529842ef75f" containerName="console" containerID="cri-o://443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5" gracePeriod=15
Jul 28 17:15:02.060593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.060572    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:15:02.232171 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232137    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232189    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232220    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232264    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdjgw\" (UniqueName: \"kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232301    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232341    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert\") pod \"87366516-387b-42e0-a8bd-b529842ef75f\" (UID: \"87366516-387b-42e0-a8bd-b529842ef75f\") "
Jul 28 17:15:02.232741 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca" (OuterVolumeSpecName: "service-ca") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:02.232741 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232731    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config" (OuterVolumeSpecName: "console-config") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:02.232869 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.232799    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:02.239930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.239885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw" (OuterVolumeSpecName: "kube-api-access-xdjgw") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "kube-api-access-xdjgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:15:02.239930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.239908    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:02.240096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.239965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87366516-387b-42e0-a8bd-b529842ef75f" (UID: "87366516-387b-42e0-a8bd-b529842ef75f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:02.333457 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333432    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjgw\" (UniqueName: \"kubernetes.io/projected/87366516-387b-42e0-a8bd-b529842ef75f-kube-api-access-xdjgw\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.333457 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333455    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.333593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333464    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-oauth-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.333593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333473    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-service-ca\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.333593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333483    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87366516-387b-42e0-a8bd-b529842ef75f-console-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.333593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.333490    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87366516-387b-42e0-a8bd-b529842ef75f-console-oauth-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:02.832237 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.832202    2573 generic.go:356] "Generic (PLEG): container finished" podID="87366516-387b-42e0-a8bd-b529842ef75f" containerID="443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5" exitCode=0
Jul 28 17:15:02.832638 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.832254    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbc9d5947-qqd54"
Jul 28 17:15:02.832638 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.832277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbc9d5947-qqd54" event={"ID":"87366516-387b-42e0-a8bd-b529842ef75f","Type":"ContainerDied","Data":"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"}
Jul 28 17:15:02.832638 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.832332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbc9d5947-qqd54" event={"ID":"87366516-387b-42e0-a8bd-b529842ef75f","Type":"ContainerDied","Data":"ba83106176f7bb4dcc03154114be2794dfc86a58a71aed1ee8eb9261e36bbda4"}
Jul 28 17:15:02.832638 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.832355    2573 scope.go:117] "RemoveContainer" containerID="443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"
Jul 28 17:15:02.840329 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.840313    2573 scope.go:117] "RemoveContainer" containerID="443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"
Jul 28 17:15:02.840572 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:15:02.840555    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5\": container with ID starting with 443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5 not found: ID does not exist" containerID="443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"
Jul 28 17:15:02.840624 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.840579    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5"} err="failed to get container status \"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5\": rpc error: code = NotFound desc = could not find container \"443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5\": container with ID starting with 443e0919324a515214075cee132c8297e411dc53a28352542949a36f74535ab5 not found: ID does not exist"
Jul 28 17:15:02.862210 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.862189    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:15:02.869641 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:02.869624    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cbc9d5947-qqd54"]
Jul 28 17:15:03.599801 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:03.599771    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87366516-387b-42e0-a8bd-b529842ef75f" path="/var/lib/kubelet/pods/87366516-387b-42e0-a8bd-b529842ef75f/volumes"
Jul 28 17:15:09.674321 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.674230    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" podUID="ee3fcca8-d7e2-4b03-a47b-2dc8035fc366" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:15:09.674321 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.674302    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72"
Jul 28 17:15:09.674873 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.674850    2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"49242da9ce835a970dee18fa674f8a8b187103afea2fe01c5e38a24ad8bd3916"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 17:15:09.674939 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.674901    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" podUID="ee3fcca8-d7e2-4b03-a47b-2dc8035fc366" containerName="service-proxy" containerID="cri-o://49242da9ce835a970dee18fa674f8a8b187103afea2fe01c5e38a24ad8bd3916" gracePeriod=30
Jul 28 17:15:09.858541 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.858513    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee3fcca8-d7e2-4b03-a47b-2dc8035fc366" containerID="49242da9ce835a970dee18fa674f8a8b187103afea2fe01c5e38a24ad8bd3916" exitCode=2
Jul 28 17:15:09.858643 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.858569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerDied","Data":"49242da9ce835a970dee18fa674f8a8b187103afea2fe01c5e38a24ad8bd3916"}
Jul 28 17:15:09.858643 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:09.858599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6674df7f-wcr72" event={"ID":"ee3fcca8-d7e2-4b03-a47b-2dc8035fc366","Type":"ContainerStarted","Data":"f152305a8d2811bec7d8573f9d3756a8e00ae6af3e2d689f70696a03cf8038fc"}
Jul 28 17:15:10.444519 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:10.444485    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:10.509823 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:10.509786    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:10.571973 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:10.571943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:13.651019 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:13.650985    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:13.722941 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:13.722897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:15.808978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.808952    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:15:15.809343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.809180    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87366516-387b-42e0-a8bd-b529842ef75f" containerName="console"
Jul 28 17:15:15.809343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.809191    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87366516-387b-42e0-a8bd-b529842ef75f" containerName="console"
Jul 28 17:15:15.809343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.809259    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87366516-387b-42e0-a8bd-b529842ef75f" containerName="console"
Jul 28 17:15:15.811888 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.811864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.822043 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.822022    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:15:15.828961 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.828941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.828967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.828997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.829018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829225 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.829103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtb4\" (UniqueName: \"kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829225 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.829182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.829225 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.829213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.865978 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.865952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:15.929728 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.929856 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.929915 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.929915 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930009 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtb4\" (UniqueName: \"kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930009 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930009 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.929974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930594 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.930572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930707 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.930679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.930830 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.930813    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.931150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.931125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.932150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.932126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.932222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.932164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.938419 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.938402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtb4\" (UniqueName: \"kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4\") pod \"console-5957dfd44b-tw949\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") " pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:15.977455 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:15.977434    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:16.120123 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:16.120036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:16.240929 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:16.240894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:15:16.243691 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:15:16.243666    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29824073_181b_47bd_b3a2_f79384023409.slice/crio-60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f WatchSource:0}: Error finding container 60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f: Status 404 returned error can't find the container with id 60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f
Jul 28 17:15:16.881618 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:16.881585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5957dfd44b-tw949" event={"ID":"29824073-181b-47bd-b3a2-f79384023409","Type":"ContainerStarted","Data":"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"}
Jul 28 17:15:16.882006 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:16.881634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5957dfd44b-tw949" event={"ID":"29824073-181b-47bd-b3a2-f79384023409","Type":"ContainerStarted","Data":"60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f"}
Jul 28 17:15:16.900314 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:16.900257    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5957dfd44b-tw949" podStartSLOduration=1.900238017 podStartE2EDuration="1.900238017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:16.898905012 +0000 UTC m=+359.826826667" watchObservedRunningTime="2026-07-28 17:15:16.900238017 +0000 UTC m=+359.828159673"
Jul 28 17:15:25.593093 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:25.593039    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:25.654243 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:25.654208    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:26.121091 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:26.121036    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:26.121253 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:26.121144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:26.125646 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:26.125625    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:26.921306 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:26.921278    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:15:26.971243 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:26.971216    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:15:27.092858 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:27.092825    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:27.193822 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:27.193743    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:27.258095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:27.258062    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.442872 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.442832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:15:30.444985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.444967    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae1727b-dacc-439b-b01d-81f6b2bdc431-metrics-certs\") pod \"network-metrics-daemon-lc55j\" (UID: \"cae1727b-dacc-439b-b01d-81f6b2bdc431\") " pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:15:30.504880 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.504856    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-24g9l\""
Jul 28 17:15:30.512377 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.512357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc55j"
Jul 28 17:15:30.637926 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.637900    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc55j"]
Jul 28 17:15:30.640293 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:15:30.640253    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae1727b_dacc_439b_b01d_81f6b2bdc431.slice/crio-c24f69583475c4fcdabd160519a6410b88e00e4ec97d1a0c24e5b9c99d6f130e WatchSource:0}: Error finding container c24f69583475c4fcdabd160519a6410b88e00e4ec97d1a0c24e5b9c99d6f130e: Status 404 returned error can't find the container with id c24f69583475c4fcdabd160519a6410b88e00e4ec97d1a0c24e5b9c99d6f130e
Jul 28 17:15:30.642297 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.642279    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:15:30.931631 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:30.931601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc55j" event={"ID":"cae1727b-dacc-439b-b01d-81f6b2bdc431","Type":"ContainerStarted","Data":"c24f69583475c4fcdabd160519a6410b88e00e4ec97d1a0c24e5b9c99d6f130e"}
Jul 28 17:15:31.935952 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:31.935915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc55j" event={"ID":"cae1727b-dacc-439b-b01d-81f6b2bdc431","Type":"ContainerStarted","Data":"c1e0b900749c4a4c33c86acc4bd3b92f4c9ade33e22ba41ed34ce9fa8cb58479"}
Jul 28 17:15:31.936352 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:31.935978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc55j" event={"ID":"cae1727b-dacc-439b-b01d-81f6b2bdc431","Type":"ContainerStarted","Data":"847694671ad82ff327319e113119ee44e86047c474d7b6862e7f92c1243c27c4"}
Jul 28 17:15:31.954666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:31.954626    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lc55j" podStartSLOduration=373.163817603 podStartE2EDuration="6m13.954612867s" totalImagesPullingTime="790.795264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:31.952952559 +0000 UTC m=+374.880874224" watchObservedRunningTime="2026-07-28 17:15:31.954612867 +0000 UTC m=+374.882534521"
Jul 28 17:15:32.082530 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:32.082495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:32.149964 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:32.149936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:33.954140 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:33.954105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:34.055836 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:34.055808    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:39.341759 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:39.341724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:39.440701 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:39.440670    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:49.964926 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:49.964880    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:51.993755 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:51.993692    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c78f98f4c-dwqnv" podUID="b7be772f-6755-4459-bb18-1da0b2dd3723" containerName="console" containerID="cri-o://98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7" gracePeriod=15
Jul 28 17:15:52.233542 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.233518    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:15:52.301405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301375    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301395    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301412    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301431    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hvpr\" (UniqueName: \"kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301465    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle\") pod \"b7be772f-6755-4459-bb18-1da0b2dd3723\" (UID: \"b7be772f-6755-4459-bb18-1da0b2dd3723\") "
Jul 28 17:15:52.301880 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301848    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:52.301880 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301864    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config" (OuterVolumeSpecName: "console-config") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:52.301972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301870    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:52.301972 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.301961    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:52.303489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.303470    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:52.303581 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.303487    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr" (OuterVolumeSpecName: "kube-api-access-7hvpr") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "kube-api-access-7hvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:15:52.303619 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.303580    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7be772f-6755-4459-bb18-1da0b2dd3723" (UID: "b7be772f-6755-4459-bb18-1da0b2dd3723"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:52.402284 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402240    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-oauth-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402284 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402278    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-oauth-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402284 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402288    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-service-ca\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402284 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402295    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-console-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402304    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvpr\" (UniqueName: \"kubernetes.io/projected/b7be772f-6755-4459-bb18-1da0b2dd3723-kube-api-access-7hvpr\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402312    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7be772f-6755-4459-bb18-1da0b2dd3723-console-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:52.402517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:52.402322    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7be772f-6755-4459-bb18-1da0b2dd3723-trusted-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:15:53.005866 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.005833    2573 generic.go:356] "Generic (PLEG): container finished" podID="b7be772f-6755-4459-bb18-1da0b2dd3723" containerID="98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7" exitCode=0
Jul 28 17:15:53.006302 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.005896    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78f98f4c-dwqnv"
Jul 28 17:15:53.006302 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.005899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78f98f4c-dwqnv" event={"ID":"b7be772f-6755-4459-bb18-1da0b2dd3723","Type":"ContainerDied","Data":"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"}
Jul 28 17:15:53.006302 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.005998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78f98f4c-dwqnv" event={"ID":"b7be772f-6755-4459-bb18-1da0b2dd3723","Type":"ContainerDied","Data":"913a636e52e2ea1a352555d4d974c73b7b559ba5e597ddb3b8fed033593d12f8"}
Jul 28 17:15:53.006302 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.006017    2573 scope.go:117] "RemoveContainer" containerID="98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"
Jul 28 17:15:53.013909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.013894    2573 scope.go:117] "RemoveContainer" containerID="98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"
Jul 28 17:15:53.014170 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:15:53.014153    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7\": container with ID starting with 98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7 not found: ID does not exist" containerID="98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"
Jul 28 17:15:53.014218 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.014177    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7"} err="failed to get container status \"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7\": rpc error: code = NotFound desc = could not find container \"98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7\": container with ID starting with 98ec278cecc73d38749d0d17130d6a422eaf0a3ae012ac851dbb1aabace1afb7 not found: ID does not exist"
Jul 28 17:15:53.027508 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.027489    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:15:53.031592 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.031571    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c78f98f4c-dwqnv"]
Jul 28 17:15:53.572329 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.572298    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:53.601384 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:15:53.601356    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7be772f-6755-4459-bb18-1da0b2dd3723" path="/var/lib/kubelet/pods/b7be772f-6755-4459-bb18-1da0b2dd3723/volumes"
Jul 28 17:16:30.016164 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:30.016128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:30.121237 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:30.121199    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:30.425781 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:30.425749    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:30.557640 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:30.557605    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:30.682181 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:30.682105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:52.551856 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:52.551763    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:52.648660 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:16:52.648621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:04.623910 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.623870    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:04.912763 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.912717    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:04.913611 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.913588    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-bt9ws"]
Jul 28 17:17:04.913864 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.913849    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7be772f-6755-4459-bb18-1da0b2dd3723" containerName="console"
Jul 28 17:17:04.913931 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.913868    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7be772f-6755-4459-bb18-1da0b2dd3723" containerName="console"
Jul 28 17:17:04.914017 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.913939    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7be772f-6755-4459-bb18-1da0b2dd3723" containerName="console"
Jul 28 17:17:04.916831 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.916814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:04.920183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.920164    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:17:04.920273 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.920197    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:17:04.921237 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.921218    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:17:04.922165 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.922146    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vtwsb\""
Jul 28 17:17:04.922254 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.922177    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:17:04.923535 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.923517    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 17:17:04.937004 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:04.936981    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-bt9ws"]
Jul 28 17:17:05.000115 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.000081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59n7f\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-kube-api-access-59n7f\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.000115 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.000120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-cabundle0\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.000317 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.000210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.101035 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.100995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.101222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.101070    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59n7f\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-kube-api-access-59n7f\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.101222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.101094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-cabundle0\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.101222 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.101154    2573 secret.go:281] references non-existent secret key: ca.crt
Jul 28 17:17:05.101222 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.101177    2573 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 17:17:05.101222 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.101187    2573 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-bt9ws: references non-existent secret key: ca.crt
Jul 28 17:17:05.101402 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.101253    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates podName:eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:05.60123739 +0000 UTC m=+468.529159249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates") pod "keda-operator-77f648d694-bt9ws" (UID: "eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f") : references non-existent secret key: ca.crt
Jul 28 17:17:05.101685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.101669    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-cabundle0\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.114244 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.114221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59n7f\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-kube-api-access-59n7f\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.114725 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.114702    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.215126 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.215012    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.598955 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.598862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.605003 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.604971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.606062 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.606028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ksfmq"]
Jul 28 17:17:05.607401 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.607384    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f-certificates\") pod \"keda-operator-77f648d694-bt9ws\" (UID: \"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f\") " pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.609057 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.609034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.611851 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.611832    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 17:17:05.619642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.619620    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ksfmq"]
Jul 28 17:17:05.660510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.660471    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.705602 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.705566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.705750 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.705617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vzq\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-kube-api-access-r5vzq\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.718472 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.718441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.806879 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.806837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vzq\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-kube-api-access-r5vzq\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.807077 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.806921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.807077 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.807012    2573 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 17:17:05.807077 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.807035    2573 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-ksfmq: secret "keda-admission-webhooks-certs" not found
Jul 28 17:17:05.807193 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:17:05.807118    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates podName:86aff0f4-1186-44b5-a3cc-2fcfb2690863 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:06.307097419 +0000 UTC m=+469.235019054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates") pod "keda-admission-56f9798bb4-ksfmq" (UID: "86aff0f4-1186-44b5-a3cc-2fcfb2690863") : secret "keda-admission-webhooks-certs" not found
Jul 28 17:17:05.819668 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.819641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vzq\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-kube-api-access-r5vzq\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:05.826472 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.826437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:05.937108 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.937071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:05.957912 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:05.957878    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-bt9ws"]
Jul 28 17:17:05.960544 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:17:05.960517    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacc5dca_fe55_4a73_8bd5_14a2a4a8a41f.slice/crio-f966580e115fcffeb8f3d2d73ee3a805c9d32b454d32249fc3724fb282ce4f8e WatchSource:0}: Error finding container f966580e115fcffeb8f3d2d73ee3a805c9d32b454d32249fc3724fb282ce4f8e: Status 404 returned error can't find the container with id f966580e115fcffeb8f3d2d73ee3a805c9d32b454d32249fc3724fb282ce4f8e
Jul 28 17:17:06.165174 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.165135    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:06.251456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.251418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-bt9ws" event={"ID":"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f","Type":"ContainerStarted","Data":"f966580e115fcffeb8f3d2d73ee3a805c9d32b454d32249fc3724fb282ce4f8e"}
Jul 28 17:17:06.270350 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.270313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:06.312059 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.312011    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:06.314420 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.314401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86aff0f4-1186-44b5-a3cc-2fcfb2690863-certificates\") pod \"keda-admission-56f9798bb4-ksfmq\" (UID: \"86aff0f4-1186-44b5-a3cc-2fcfb2690863\") " pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:06.366756 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.366720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:06.522526 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.522436    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:06.650081 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:06.650035    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ksfmq"]
Jul 28 17:17:06.652857 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:17:06.652829    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aff0f4_1186_44b5_a3cc_2fcfb2690863.slice/crio-6ff01fb7e2b4463e8fa39e4fb1e1dabf42da1e7d09aae2d8c86fb3d51f4df0a0 WatchSource:0}: Error finding container 6ff01fb7e2b4463e8fa39e4fb1e1dabf42da1e7d09aae2d8c86fb3d51f4df0a0: Status 404 returned error can't find the container with id 6ff01fb7e2b4463e8fa39e4fb1e1dabf42da1e7d09aae2d8c86fb3d51f4df0a0
Jul 28 17:17:07.257765 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:07.257687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq" event={"ID":"86aff0f4-1186-44b5-a3cc-2fcfb2690863","Type":"ContainerStarted","Data":"6ff01fb7e2b4463e8fa39e4fb1e1dabf42da1e7d09aae2d8c86fb3d51f4df0a0"}
Jul 28 17:17:08.272625 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:08.272555    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:11.275828 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.275794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-bt9ws" event={"ID":"eacc5dca-fe55-4a73-8bd5-14a2a4a8a41f","Type":"ContainerStarted","Data":"324dcb4bce674fb4024c08fd5dbbe0798938b42b6778ed705754575d6cf859e3"}
Jul 28 17:17:11.276276 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.275921    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:11.277180 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.277155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq" event={"ID":"86aff0f4-1186-44b5-a3cc-2fcfb2690863","Type":"ContainerStarted","Data":"f6bd076303aa4bf00f154167b0e94a833526f40e8c516ebf0624717cd04fc733"}
Jul 28 17:17:11.277293 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.277277    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:17:11.297889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.297840    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-bt9ws" podStartSLOduration=2.814067617 podStartE2EDuration="7.297825947s" totalImagesPullingTime="4.48375833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:11.29683317 +0000 UTC m=+474.224754826" watchObservedRunningTime="2026-07-28 17:17:11.297825947 +0000 UTC m=+474.225747603"
Jul 28 17:17:11.318611 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:11.318567    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq" podStartSLOduration=2.533015399 podStartE2EDuration="6.318552001s" totalImagesPullingTime="3.785536602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:11.317756388 +0000 UTC m=+474.245678044" watchObservedRunningTime="2026-07-28 17:17:11.318552001 +0000 UTC m=+474.246473656"
Jul 28 17:17:32.282594 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:32.282557    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-bt9ws"
Jul 28 17:17:32.282956 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:17:32.282629    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-ksfmq"
Jul 28 17:18:04.713633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:04.713590    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:11.408719 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:11.408635    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.039099 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.039063    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.113342 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.113305    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:18:22.115680 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.115651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.118626 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.118605    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:18:22.118835 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.118821    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-psntj\""
Jul 28 17:18:22.120305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.120285    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:18:22.120407 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.120290    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:18:22.129215 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.129189    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:18:22.153687 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.153653    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4mpbj"]
Jul 28 17:18:22.155753 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.155736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.158633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.158613    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q2x2w\""
Jul 28 17:18:22.158746 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.158666    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:18:22.165536 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.165513    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4mpbj"]
Jul 28 17:18:22.275598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.275556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e12c82f-e6d5-4447-90ad-6684de53cd4e-data\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.275784 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.275604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ztm\" (UniqueName: \"kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.275784 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.275681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.275784 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.275724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4bhw\" (UniqueName: \"kubernetes.io/projected/8e12c82f-e6d5-4447-90ad-6684de53cd4e-kube-api-access-f4bhw\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.376642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.376552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.376642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.376594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4bhw\" (UniqueName: \"kubernetes.io/projected/8e12c82f-e6d5-4447-90ad-6684de53cd4e-kube-api-access-f4bhw\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.376642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.376641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e12c82f-e6d5-4447-90ad-6684de53cd4e-data\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.376874 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.376664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ztm\" (UniqueName: \"kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.376874 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:18:22.376700    2573 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 17:18:22.376874 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:18:22.376782    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert podName:e6c07bea-4171-4c37-a25e-a8c323cc899d nodeName:}" failed. No retries permitted until 2026-07-28 17:18:22.876743375 +0000 UTC m=+545.804665008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert") pod "kserve-controller-manager-cd4d44c67-pcwk9" (UID: "e6c07bea-4171-4c37-a25e-a8c323cc899d") : secret "kserve-webhook-server-cert" not found
Jul 28 17:18:22.377106 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.377078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8e12c82f-e6d5-4447-90ad-6684de53cd4e-data\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.387011 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.386984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ztm\" (UniqueName: \"kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.387011 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.387010    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4bhw\" (UniqueName: \"kubernetes.io/projected/8e12c82f-e6d5-4447-90ad-6684de53cd4e-kube-api-access-f4bhw\") pod \"seaweedfs-64568bcd49-4mpbj\" (UID: \"8e12c82f-e6d5-4447-90ad-6684de53cd4e\") " pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.464170 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.464137    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:22.591096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.591073    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4mpbj"]
Jul 28 17:18:22.593668 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:18:22.593640    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e12c82f_e6d5_4447_90ad_6684de53cd4e.slice/crio-9644ea2d34c1dddcf0d184dbec327337020b1a52589ac3fad086cdeb72dce7ca WatchSource:0}: Error finding container 9644ea2d34c1dddcf0d184dbec327337020b1a52589ac3fad086cdeb72dce7ca: Status 404 returned error can't find the container with id 9644ea2d34c1dddcf0d184dbec327337020b1a52589ac3fad086cdeb72dce7ca
Jul 28 17:18:22.880925 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.880876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:22.883435 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:22.883411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") pod \"kserve-controller-manager-cd4d44c67-pcwk9\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") " pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:23.025207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:23.025174    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:23.161441 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:23.161407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:18:23.164186 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:18:23.164164    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c07bea_4171_4c37_a25e_a8c323cc899d.slice/crio-aaea376189df494f856dc95514aa2859ab4aed9b91c8cfdec9e8ff9357fb9d81 WatchSource:0}: Error finding container aaea376189df494f856dc95514aa2859ab4aed9b91c8cfdec9e8ff9357fb9d81: Status 404 returned error can't find the container with id aaea376189df494f856dc95514aa2859ab4aed9b91c8cfdec9e8ff9357fb9d81
Jul 28 17:18:23.483294 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:23.483209    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.533473 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:23.533421    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4mpbj" event={"ID":"8e12c82f-e6d5-4447-90ad-6684de53cd4e","Type":"ContainerStarted","Data":"9644ea2d34c1dddcf0d184dbec327337020b1a52589ac3fad086cdeb72dce7ca"}
Jul 28 17:18:23.534695 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:23.534657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" event={"ID":"e6c07bea-4171-4c37-a25e-a8c323cc899d","Type":"ContainerStarted","Data":"aaea376189df494f856dc95514aa2859ab4aed9b91c8cfdec9e8ff9357fb9d81"}
Jul 28 17:18:27.552244 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.552205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" event={"ID":"e6c07bea-4171-4c37-a25e-a8c323cc899d","Type":"ContainerStarted","Data":"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"}
Jul 28 17:18:27.552686 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.552310    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:18:27.553661 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.553636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4mpbj" event={"ID":"8e12c82f-e6d5-4447-90ad-6684de53cd4e","Type":"ContainerStarted","Data":"49b99583bbe78de949c7ed95eb07be3c52b7fd3eceda616c993482dcde54e47b"}
Jul 28 17:18:27.553747 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.553686    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:27.571680 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.571635    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" podStartSLOduration=2.083165516 podStartE2EDuration="5.571621185s" totalImagesPullingTime="3.488455669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:27.570715215 +0000 UTC m=+550.498636871" watchObservedRunningTime="2026-07-28 17:18:27.571621185 +0000 UTC m=+550.499542839"
Jul 28 17:18:27.588973 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:27.588930    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4mpbj" podStartSLOduration=1.496748736 podStartE2EDuration="5.588913012s" totalImagesPullingTime="4.092164276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:27.588278378 +0000 UTC m=+550.516200044" watchObservedRunningTime="2026-07-28 17:18:27.588913012 +0000 UTC m=+550.516834667"
Jul 28 17:18:33.559494 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:33.559457    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4mpbj"
Jul 28 17:18:47.906726 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:47.906695    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:48.007870 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:48.007838    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:49.654642 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:49.654604    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:49.716438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:49.716405    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:58.561807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:18:58.561778    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:19:04.589999 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.589965    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:19:04.590417 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.590172    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" podUID="e6c07bea-4171-4c37-a25e-a8c323cc899d" containerName="manager" containerID="cri-o://ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b" gracePeriod=10
Jul 28 17:19:04.613807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.613782    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-f9jbv"]
Jul 28 17:19:04.615896 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.615880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.627318 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.627289    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-f9jbv"]
Jul 28 17:19:04.683739 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.683704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde3300b-b272-4120-a941-1ad3b1598a39-cert\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.683883 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.683775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqwn\" (UniqueName: \"kubernetes.io/projected/bde3300b-b272-4120-a941-1ad3b1598a39-kube-api-access-9pqwn\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.784572 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.784533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqwn\" (UniqueName: \"kubernetes.io/projected/bde3300b-b272-4120-a941-1ad3b1598a39-kube-api-access-9pqwn\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.784757 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.784618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde3300b-b272-4120-a941-1ad3b1598a39-cert\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.786945 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.786920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde3300b-b272-4120-a941-1ad3b1598a39-cert\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.794165 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.794138    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqwn\" (UniqueName: \"kubernetes.io/projected/bde3300b-b272-4120-a941-1ad3b1598a39-kube-api-access-9pqwn\") pod \"kserve-controller-manager-cd4d44c67-f9jbv\" (UID: \"bde3300b-b272-4120-a941-1ad3b1598a39\") " pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.829581 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.829557    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:19:04.885262 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.885185    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ztm\" (UniqueName: \"kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm\") pod \"e6c07bea-4171-4c37-a25e-a8c323cc899d\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") "
Jul 28 17:19:04.885405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.885284    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") pod \"e6c07bea-4171-4c37-a25e-a8c323cc899d\" (UID: \"e6c07bea-4171-4c37-a25e-a8c323cc899d\") "
Jul 28 17:19:04.887343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.887313    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert" (OuterVolumeSpecName: "cert") pod "e6c07bea-4171-4c37-a25e-a8c323cc899d" (UID: "e6c07bea-4171-4c37-a25e-a8c323cc899d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:04.887456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.887409    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm" (OuterVolumeSpecName: "kube-api-access-87ztm") pod "e6c07bea-4171-4c37-a25e-a8c323cc899d" (UID: "e6c07bea-4171-4c37-a25e-a8c323cc899d"). InnerVolumeSpecName "kube-api-access-87ztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:04.967074 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.967013    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:04.985813 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.985783    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ztm\" (UniqueName: \"kubernetes.io/projected/e6c07bea-4171-4c37-a25e-a8c323cc899d-kube-api-access-87ztm\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:04.985813 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:04.985814    2573 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6c07bea-4171-4c37-a25e-a8c323cc899d-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:05.096471 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.096447    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-f9jbv"]
Jul 28 17:19:05.098629 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:19:05.098603    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde3300b_b272_4120_a941_1ad3b1598a39.slice/crio-c6187933a0bc269fa40be57899224c53ab76c79e1161371fbcd4c124403fe197 WatchSource:0}: Error finding container c6187933a0bc269fa40be57899224c53ab76c79e1161371fbcd4c124403fe197: Status 404 returned error can't find the container with id c6187933a0bc269fa40be57899224c53ab76c79e1161371fbcd4c124403fe197
Jul 28 17:19:05.680306 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.680275    2573 generic.go:356] "Generic (PLEG): container finished" podID="e6c07bea-4171-4c37-a25e-a8c323cc899d" containerID="ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b" exitCode=0
Jul 28 17:19:05.680647 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.680320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" event={"ID":"e6c07bea-4171-4c37-a25e-a8c323cc899d","Type":"ContainerDied","Data":"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"}
Jul 28 17:19:05.680647 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.680345    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9"
Jul 28 17:19:05.680647 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.680397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-pcwk9" event={"ID":"e6c07bea-4171-4c37-a25e-a8c323cc899d","Type":"ContainerDied","Data":"aaea376189df494f856dc95514aa2859ab4aed9b91c8cfdec9e8ff9357fb9d81"}
Jul 28 17:19:05.680647 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.680429    2573 scope.go:117] "RemoveContainer" containerID="ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"
Jul 28 17:19:05.681588 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.681552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv" event={"ID":"bde3300b-b272-4120-a941-1ad3b1598a39","Type":"ContainerStarted","Data":"c6187933a0bc269fa40be57899224c53ab76c79e1161371fbcd4c124403fe197"}
Jul 28 17:19:05.688349 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.688330    2573 scope.go:117] "RemoveContainer" containerID="ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"
Jul 28 17:19:05.688629 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:19:05.688609    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b\": container with ID starting with ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b not found: ID does not exist" containerID="ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"
Jul 28 17:19:05.688681 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.688638    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b"} err="failed to get container status \"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b\": rpc error: code = NotFound desc = could not find container \"ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b\": container with ID starting with ab446033cfa051e25f51630ea7198ba944a2785f69afcdbe0e48e1f12d5de53b not found: ID does not exist"
Jul 28 17:19:05.718582 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.718545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:19:05.722502 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:05.722469    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-pcwk9"]
Jul 28 17:19:06.687605 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:06.687570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv" event={"ID":"bde3300b-b272-4120-a941-1ad3b1598a39","Type":"ContainerStarted","Data":"f2e0586387e6f006662e90fc9818abee2888bf0c113f7b5983ec6b84025bc96b"}
Jul 28 17:19:06.687980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:06.687629    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:06.707015 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:06.706969    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv" podStartSLOduration=2.169556121 podStartE2EDuration="2.706954761s" totalImagesPullingTime="537.39864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:06.705490759 +0000 UTC m=+589.633412416" watchObservedRunningTime="2026-07-28 17:19:06.706954761 +0000 UTC m=+589.634876417"
Jul 28 17:19:07.600599 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:07.600556    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c07bea-4171-4c37-a25e-a8c323cc899d" path="/var/lib/kubelet/pods/e6c07bea-4171-4c37-a25e-a8c323cc899d/volumes"
Jul 28 17:19:13.525278 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:13.525236    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:13.627095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:13.627037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:21.484666 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.484627    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-677f74f889-rzf5t"]
Jul 28 17:19:21.488432 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.484900    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c07bea-4171-4c37-a25e-a8c323cc899d" containerName="manager"
Jul 28 17:19:21.488432 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.484912    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c07bea-4171-4c37-a25e-a8c323cc899d" containerName="manager"
Jul 28 17:19:21.488432 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.484978    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c07bea-4171-4c37-a25e-a8c323cc899d" containerName="manager"
Jul 28 17:19:21.489730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.489716    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.499183 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.499157    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677f74f889-rzf5t"]
Jul 28 17:19:21.536125 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.536089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:21.598268 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.598239    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:21.616271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftkv\" (UniqueName: \"kubernetes.io/projected/7e297f07-650a-444d-818f-d7d82eac2a9a-kube-api-access-hftkv\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616389 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-trusted-ca-bundle\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616389 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-oauth-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616459 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-oauth-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616459 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-service-ca\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-console-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.616523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.616494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717345 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftkv\" (UniqueName: \"kubernetes.io/projected/7e297f07-650a-444d-818f-d7d82eac2a9a-kube-api-access-hftkv\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717505 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-trusted-ca-bundle\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717505 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-oauth-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717505 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-oauth-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717505 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-service-ca\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717505 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-console-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.717754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.717517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.718291 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.718267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-service-ca\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.718291 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.718280    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-oauth-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.718434 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.718272    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-trusted-ca-bundle\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.718434 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.718298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e297f07-650a-444d-818f-d7d82eac2a9a-console-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.720403 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.720386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-serving-cert\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.720517 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.720499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e297f07-650a-444d-818f-d7d82eac2a9a-console-oauth-config\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.728007 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.727986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftkv\" (UniqueName: \"kubernetes.io/projected/7e297f07-650a-444d-818f-d7d82eac2a9a-kube-api-access-hftkv\") pod \"console-677f74f889-rzf5t\" (UID: \"7e297f07-650a-444d-818f-d7d82eac2a9a\") " pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.798854 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.798772    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:21.935716 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:21.935666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677f74f889-rzf5t"]
Jul 28 17:19:21.938604 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:19:21.938565    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e297f07_650a_444d_818f_d7d82eac2a9a.slice/crio-f9d3738377d233ccfef15bb36ae272fbaba2e4fbda253622c2ef6393fda5b095 WatchSource:0}: Error finding container f9d3738377d233ccfef15bb36ae272fbaba2e4fbda253622c2ef6393fda5b095: Status 404 returned error can't find the container with id f9d3738377d233ccfef15bb36ae272fbaba2e4fbda253622c2ef6393fda5b095
Jul 28 17:19:22.745381 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:22.745346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677f74f889-rzf5t" event={"ID":"7e297f07-650a-444d-818f-d7d82eac2a9a","Type":"ContainerStarted","Data":"59d4db50fb92dbd995c25c8a2a35b56ce8500711f0dced2452ec6fe15c37b6e5"}
Jul 28 17:19:22.745734 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:22.745394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677f74f889-rzf5t" event={"ID":"7e297f07-650a-444d-818f-d7d82eac2a9a","Type":"ContainerStarted","Data":"f9d3738377d233ccfef15bb36ae272fbaba2e4fbda253622c2ef6393fda5b095"}
Jul 28 17:19:22.766556 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:22.766509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-677f74f889-rzf5t" podStartSLOduration=1.7664957860000001 podStartE2EDuration="1.766495786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:22.765738984 +0000 UTC m=+605.693660640" watchObservedRunningTime="2026-07-28 17:19:22.766495786 +0000 UTC m=+605.694417441"
Jul 28 17:19:24.303998 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:24.303943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:31.799385 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:31.799343    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:31.799782 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:31.799411    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:31.804271 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:31.804243    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:32.786095 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:32.786040    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-677f74f889-rzf5t"
Jul 28 17:19:32.838015 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:32.837980    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:19:32.942161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:32.942126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:33.003093 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:33.003034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:33.064977 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:33.064891    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:37.695918 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:37.695888    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-f9jbv"
Jul 28 17:19:38.757714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.757676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:38.830868 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.830834    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tcs7p"]
Jul 28 17:19:38.834040 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.834023    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:38.837405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.837386    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-m2bhz\""
Jul 28 17:19:38.837599 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.837585    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:19:38.846774 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.846751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tcs7p"]
Jul 28 17:19:38.851808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.851788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:38.851895 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.851815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7zz\" (UniqueName: \"kubernetes.io/projected/73f5709b-4a02-445d-bf1b-1e88f2dab521-kube-api-access-pt7zz\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:38.855092 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.855065    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:38.952198 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.952164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:38.952388 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.952209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7zz\" (UniqueName: \"kubernetes.io/projected/73f5709b-4a02-445d-bf1b-1e88f2dab521-kube-api-access-pt7zz\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:38.952388 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:19:38.952282    2573 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 17:19:38.952388 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:19:38.952348    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert podName:73f5709b-4a02-445d-bf1b-1e88f2dab521 nodeName:}" failed. No retries permitted until 2026-07-28 17:19:39.45233141 +0000 UTC m=+622.380253048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert") pod "odh-model-controller-75d9cdbddf-tcs7p" (UID: "73f5709b-4a02-445d-bf1b-1e88f2dab521") : secret "odh-model-controller-webhook-cert" not found
Jul 28 17:19:38.952847 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.952822    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:38.964583 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:38.964551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7zz\" (UniqueName: \"kubernetes.io/projected/73f5709b-4a02-445d-bf1b-1e88f2dab521-kube-api-access-pt7zz\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:39.061042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.060963    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:39.455132 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.455104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:39.457462 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.457443    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73f5709b-4a02-445d-bf1b-1e88f2dab521-cert\") pod \"odh-model-controller-75d9cdbddf-tcs7p\" (UID: \"73f5709b-4a02-445d-bf1b-1e88f2dab521\") " pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:39.675990 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.675950    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:39.744099 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.744011    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:39.746547 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.746512    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:39.870754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:39.870725    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tcs7p"]
Jul 28 17:19:39.874747 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:19:39.874721    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f5709b_4a02_445d_bf1b_1e88f2dab521.slice/crio-22c7a738969007a18eecb3b53e787bdbf162dbc7ee612de70b3d479d9cc1a31a WatchSource:0}: Error finding container 22c7a738969007a18eecb3b53e787bdbf162dbc7ee612de70b3d479d9cc1a31a: Status 404 returned error can't find the container with id 22c7a738969007a18eecb3b53e787bdbf162dbc7ee612de70b3d479d9cc1a31a
Jul 28 17:19:40.812323 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:40.812272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p" event={"ID":"73f5709b-4a02-445d-bf1b-1e88f2dab521","Type":"ContainerStarted","Data":"22c7a738969007a18eecb3b53e787bdbf162dbc7ee612de70b3d479d9cc1a31a"}
Jul 28 17:19:42.821669 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:42.821631    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p" event={"ID":"73f5709b-4a02-445d-bf1b-1e88f2dab521","Type":"ContainerStarted","Data":"8e16ac0de101775efed94b446738331d7bd5d35574f7241168228a499ff3b0fe"}
Jul 28 17:19:42.822077 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:42.821773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:42.840588 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:42.840538    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p" podStartSLOduration=2.326342514 podStartE2EDuration="4.840523218s" totalImagesPullingTime="2.514180704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:42.839466343 +0000 UTC m=+625.767388002" watchObservedRunningTime="2026-07-28 17:19:42.840523218 +0000 UTC m=+625.768444874"
Jul 28 17:19:53.827802 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:53.827768    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-tcs7p"
Jul 28 17:19:57.858276 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:57.858214    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5957dfd44b-tw949" podUID="29824073-181b-47bd-b3a2-f79384023409" containerName="console" containerID="cri-o://4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160" gracePeriod=15
Jul 28 17:19:58.099569 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.099540    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:19:58.204860 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.204826    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.204869    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.204887    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.204915    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.204960    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205011    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqtb4\" (UniqueName: \"kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205278 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205071    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert\") pod \"29824073-181b-47bd-b3a2-f79384023409\" (UID: \"29824073-181b-47bd-b3a2-f79384023409\") "
Jul 28 17:19:58.205400 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205372    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:58.205470 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205447    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config" (OuterVolumeSpecName: "console-config") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:58.205596 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205566    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca" (OuterVolumeSpecName: "service-ca") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:58.205650 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.205610    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:58.207270 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.207239    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4" (OuterVolumeSpecName: "kube-api-access-sqtb4") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "kube-api-access-sqtb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:58.207378 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.207280    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:58.207378 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.207296    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "29824073-181b-47bd-b3a2-f79384023409" (UID: "29824073-181b-47bd-b3a2-f79384023409"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:58.306578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306540    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtb4\" (UniqueName: \"kubernetes.io/projected/29824073-181b-47bd-b3a2-f79384023409-kube-api-access-sqtb4\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306570    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306580    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-trusted-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306589    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-console-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306821 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306598    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-oauth-serving-cert\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306821 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306606    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29824073-181b-47bd-b3a2-f79384023409-service-ca\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.306821 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.306614    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29824073-181b-47bd-b3a2-f79384023409-console-oauth-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:19:58.883797 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.883751    2573 generic.go:356] "Generic (PLEG): container finished" podID="29824073-181b-47bd-b3a2-f79384023409" containerID="4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160" exitCode=0
Jul 28 17:19:58.883797 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.883797    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5957dfd44b-tw949"
Jul 28 17:19:58.884287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.883829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5957dfd44b-tw949" event={"ID":"29824073-181b-47bd-b3a2-f79384023409","Type":"ContainerDied","Data":"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"}
Jul 28 17:19:58.884287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.883887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5957dfd44b-tw949" event={"ID":"29824073-181b-47bd-b3a2-f79384023409","Type":"ContainerDied","Data":"60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f"}
Jul 28 17:19:58.884287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.883909    2573 scope.go:117] "RemoveContainer" containerID="4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"
Jul 28 17:19:58.892258 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.892240    2573 scope.go:117] "RemoveContainer" containerID="4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"
Jul 28 17:19:58.892541 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:19:58.892515    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160\": container with ID starting with 4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160 not found: ID does not exist" containerID="4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"
Jul 28 17:19:58.892633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.892551    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160"} err="failed to get container status \"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160\": rpc error: code = NotFound desc = could not find container \"4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160\": container with ID starting with 4f83222e84654106b69c3f4d8016ce77193e0984246fba4799bc8fad38aa9160 not found: ID does not exist"
Jul 28 17:19:58.911258 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:19:58.911232    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29824073_181b_47bd_b3a2_f79384023409.slice/crio-60a52e29c5451ff57d9c27d3d93d600231f97e22ba68a3a587d5ecafd6af2e9f\": RecentStats: unable to find data in memory cache]"
Jul 28 17:19:58.916198 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.916177    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:19:58.924228 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:58.924207    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5957dfd44b-tw949"]
Jul 28 17:19:59.600679 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:19:59.600646    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29824073-181b-47bd-b3a2-f79384023409" path="/var/lib/kubelet/pods/29824073-181b-47bd-b3a2-f79384023409/volumes"
Jul 28 17:20:13.617992 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:20:13.617948    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:25.512435 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:20:25.512395    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:25.612469 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:20:25.612432    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:19.623735 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:19.623640    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:19.729159 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:19.729120    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:31.589957 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.589920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:31.675230 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.675192    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:21:31.675475 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.675462    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29824073-181b-47bd-b3a2-f79384023409" containerName="console"
Jul 28 17:21:31.675534 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.675476    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="29824073-181b-47bd-b3a2-f79384023409" containerName="console"
Jul 28 17:21:31.675534 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.675517    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="29824073-181b-47bd-b3a2-f79384023409" containerName="console"
Jul 28 17:21:31.678415 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.678398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.681353 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.681322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b94d0-predictor-serving-cert\""
Jul 28 17:21:31.681589 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.681570    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:21:31.681659 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.681633    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b94d0-kube-rbac-proxy-sar-config\""
Jul 28 17:21:31.682956 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.682929    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-895bt\""
Jul 28 17:21:31.683077 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.683035    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:21:31.690543 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.690522    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:21:31.794771 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.794728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.794909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.794777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.794909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.794812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zf6\" (UniqueName: \"kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.863156 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.863065    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:31.895478 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.895455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.895593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.895487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.895593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.895530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zf6\" (UniqueName: \"kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.895723 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:21:31.895607    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-serving-cert: secret "success-200-isvc-b94d0-predictor-serving-cert" not found
Jul 28 17:21:31.895723 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:21:31.895667    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls podName:5ff3495b-f4af-49c3-be73-58aa5137a2bb nodeName:}" failed. No retries permitted until 2026-07-28 17:21:32.395648358 +0000 UTC m=+735.323570020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls") pod "success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" (UID: "5ff3495b-f4af-49c3-be73-58aa5137a2bb") : secret "success-200-isvc-b94d0-predictor-serving-cert" not found
Jul 28 17:21:31.896175 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.896157    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.909318 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.909286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zf6\" (UniqueName: \"kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:31.986862 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:31.986824    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.076477 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.076440    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.155202 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.155134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.251958 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.251927    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.328102 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.328068    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.398633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.398601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:32.401084 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.401040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") pod \"success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") " pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:32.429170 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.429141    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:32.588826 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.588794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:32.724849 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.724822    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:21:32.727436 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:21:32.727410    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff3495b_f4af_49c3_be73_58aa5137a2bb.slice/crio-e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13 WatchSource:0}: Error finding container e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13: Status 404 returned error can't find the container with id e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13
Jul 28 17:21:32.729257 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:32.729241    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:21:33.218499 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:33.218401    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerStarted","Data":"e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13"}
Jul 28 17:21:44.269551 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:44.269520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerStarted","Data":"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"}
Jul 28 17:21:47.284014 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:47.283981    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerStarted","Data":"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"}
Jul 28 17:21:47.284446 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:47.284155    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:47.284446 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:47.284216    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:47.285471 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:47.285445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:48.287785 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:48.287739    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:53.292558 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:53.292529    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:21:53.293109 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:53.292940    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:53.315316 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:21:53.315264    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podStartSLOduration=8.624140388 podStartE2EDuration="22.315249794s" totalImagesPullingTime="13.691109406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:47.309278543 +0000 UTC m=+750.237200199" watchObservedRunningTime="2026-07-28 17:21:53.315249794 +0000 UTC m=+756.243171449"
Jul 28 17:22:03.293251 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:03.293212    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:22:13.293602 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:13.293560    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:22:23.293886 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:23.293786    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:22:33.294199 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:33.294165    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:22:51.679101 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.679004    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:22:51.681177 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.681159    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:51.684102 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.684083    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b94d0-serving-cert\""
Jul 28 17:22:51.684200 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.684083    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b94d0-kube-rbac-proxy-sar-config\""
Jul 28 17:22:51.687874 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.687796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:51.691281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.691252    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:22:51.713418 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.713393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:51.713523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.713456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:51.814672 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.814635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:51.814819 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.814704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:51.814819 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:22:51.814800    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b94d0-serving-cert: secret "switch-graph-b94d0-serving-cert" not found
Jul 28 17:22:51.814886 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:22:51.814858    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls podName:e41d297a-7594-472f-8af9-8404ff789851 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:52.314842913 +0000 UTC m=+815.242764547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls") pod "switch-graph-b94d0-647d855485-ztlwt" (UID: "e41d297a-7594-472f-8af9-8404ff789851") : secret "switch-graph-b94d0-serving-cert" not found
Jul 28 17:22:51.815284 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:51.815264    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:52.161092 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:52.161039    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:52.318387 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:52.318357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:52.320748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:52.320726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") pod \"switch-graph-b94d0-647d855485-ztlwt\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") " pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:52.592357 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:52.592268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:52.719620 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:52.719501    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:22:52.722265 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:22:52.722235    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41d297a_7594_472f_8af9_8404ff789851.slice/crio-58bd6409303dcc397b8570949f6be5800c7ab960b41ab4c243db5c0c263dabe7 WatchSource:0}: Error finding container 58bd6409303dcc397b8570949f6be5800c7ab960b41ab4c243db5c0c263dabe7: Status 404 returned error can't find the container with id 58bd6409303dcc397b8570949f6be5800c7ab960b41ab4c243db5c0c263dabe7
Jul 28 17:22:53.521590 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:53.521553    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" event={"ID":"e41d297a-7594-472f-8af9-8404ff789851","Type":"ContainerStarted","Data":"58bd6409303dcc397b8570949f6be5800c7ab960b41ab4c243db5c0c263dabe7"}
Jul 28 17:22:54.527037 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:54.527000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" event={"ID":"e41d297a-7594-472f-8af9-8404ff789851","Type":"ContainerStarted","Data":"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"}
Jul 28 17:22:54.527464 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:54.527132    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:22:54.548388 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:22:54.548344    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podStartSLOduration=1.815865784 podStartE2EDuration="3.54833005s" totalImagesPullingTime="1.732464266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:54.548307357 +0000 UTC m=+817.476229013" watchObservedRunningTime="2026-07-28 17:22:54.54833005 +0000 UTC m=+817.476251706"
Jul 28 17:23:00.536885 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:00.536854    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:23:01.862551 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:01.862518    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:23:01.862907 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:01.862729    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" containerID="cri-o://e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f" gracePeriod=30
Jul 28 17:23:01.892985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:01.892943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.004151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.004121    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:23:02.004370 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.004347    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" containerID="cri-o://e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7" gracePeriod=30
Jul 28 17:23:02.004467 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.004393    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kube-rbac-proxy" containerID="cri-o://a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc" gracePeriod=30
Jul 28 17:23:02.067070 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.067026    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.072142 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.072115    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:23:02.074476 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.074458    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.077453 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.077437    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-391b7-predictor-serving-cert\""
Jul 28 17:23:02.077820 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.077803    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-391b7-kube-rbac-proxy-sar-config\""
Jul 28 17:23:02.086581 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.086556    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:23:02.145113 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.145079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.201439 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.201410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.201439 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.201440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.201605 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.201474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmg4m\" (UniqueName: \"kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.221033 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.221007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.302764 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.302727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmg4m\" (UniqueName: \"kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.302962 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.302828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.302962 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.302859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.303148 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:02.303075    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-391b7-predictor-serving-cert: secret "success-200-isvc-391b7-predictor-serving-cert" not found
Jul 28 17:23:02.303148 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:02.303136    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls podName:01b01629-5732-466b-809b-4fd5067df411 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:02.803119081 +0000 UTC m=+825.731040715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls") pod "success-200-isvc-391b7-predictor-888b7969f-kw99r" (UID: "01b01629-5732-466b-809b-4fd5067df411") : secret "success-200-isvc-391b7-predictor-serving-cert" not found
Jul 28 17:23:02.303523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.303506    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.314060 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.314023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmg4m\" (UniqueName: \"kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.354277 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.354241    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.499917 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.499876    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:02.563543 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.563513    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerID="a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc" exitCode=2
Jul 28 17:23:02.563685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.563565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerDied","Data":"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"}
Jul 28 17:23:02.807624 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.807538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.810063 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.810028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") pod \"success-200-isvc-391b7-predictor-888b7969f-kw99r\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") " pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:02.984818 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:02.984781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:03.111549 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.111495    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:23:03.113754 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:23:03.113727    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b01629_5732_466b_809b_4fd5067df411.slice/crio-e08ff005f24f272559c23394639138731811e7923166b276f819ce10a0cb66b0 WatchSource:0}: Error finding container e08ff005f24f272559c23394639138731811e7923166b276f819ce10a0cb66b0: Status 404 returned error can't find the container with id e08ff005f24f272559c23394639138731811e7923166b276f819ce10a0cb66b0
Jul 28 17:23:03.288219 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.288176    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 17:23:03.293744 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.293720    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:23:03.568939 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.568901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerStarted","Data":"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"}
Jul 28 17:23:03.569152 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.568966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerStarted","Data":"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"}
Jul 28 17:23:03.569152 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.568993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerStarted","Data":"e08ff005f24f272559c23394639138731811e7923166b276f819ce10a0cb66b0"}
Jul 28 17:23:03.569152 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.569025    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:03.587818 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:03.587769    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podStartSLOduration=1.5877533449999999 podStartE2EDuration="1.587753345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:03.58752416 +0000 UTC m=+826.515445817" watchObservedRunningTime="2026-07-28 17:23:03.587753345 +0000 UTC m=+826.515675001"
Jul 28 17:23:04.572243 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:04.572214    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:04.573522 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:04.573495    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:04.955247 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:04.955225    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:23:05.025244 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.025208    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") pod \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") "
Jul 28 17:23:05.025244 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.025255    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55zf6\" (UniqueName: \"kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6\") pod \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") "
Jul 28 17:23:05.025474 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.025383    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config\") pod \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\" (UID: \"5ff3495b-f4af-49c3-be73-58aa5137a2bb\") "
Jul 28 17:23:05.025715 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.025687    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ff3495b-f4af-49c3-be73-58aa5137a2bb" (UID: "5ff3495b-f4af-49c3-be73-58aa5137a2bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:05.027350 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.027329    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6" (OuterVolumeSpecName: "kube-api-access-55zf6") pod "5ff3495b-f4af-49c3-be73-58aa5137a2bb" (UID: "5ff3495b-f4af-49c3-be73-58aa5137a2bb"). InnerVolumeSpecName "kube-api-access-55zf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:23:05.027447 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.027429    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ff3495b-f4af-49c3-be73-58aa5137a2bb" (UID: "5ff3495b-f4af-49c3-be73-58aa5137a2bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:05.126572 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.126512    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff3495b-f4af-49c3-be73-58aa5137a2bb-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:23:05.126572 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.126535    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zf6\" (UniqueName: \"kubernetes.io/projected/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-api-access-55zf6\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:23:05.126572 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.126545    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff3495b-f4af-49c3-be73-58aa5137a2bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:23:05.535523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.535481    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:05.577042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577013    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerID="e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7" exitCode=0
Jul 28 17:23:05.577489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577067    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"
Jul 28 17:23:05.577489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerDied","Data":"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"}
Jul 28 17:23:05.577489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7" event={"ID":"5ff3495b-f4af-49c3-be73-58aa5137a2bb","Type":"ContainerDied","Data":"e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13"}
Jul 28 17:23:05.577489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577211    2573 scope.go:117] "RemoveContainer" containerID="a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"
Jul 28 17:23:05.577686 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.577522    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:05.594538 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.594518    2573 scope.go:117] "RemoveContainer" containerID="e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"
Jul 28 17:23:05.601592 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:05.601567    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff3495b_f4af_49c3_be73_58aa5137a2bb.slice/crio-e702fcd0b50d755f2f94be7956f779461df50c0892527185411a066f2eb75c13\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff3495b_f4af_49c3_be73_58aa5137a2bb.slice\": RecentStats: unable to find data in memory cache]"
Jul 28 17:23:05.602431 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.602410    2573 scope.go:117] "RemoveContainer" containerID="a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"
Jul 28 17:23:05.602727 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:05.602706    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc\": container with ID starting with a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc not found: ID does not exist" containerID="a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"
Jul 28 17:23:05.602814 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.602740    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc"} err="failed to get container status \"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc\": rpc error: code = NotFound desc = could not find container \"a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc\": container with ID starting with a796ab63009de350a2c4868a920d6bd1537eb27518b5975ed78edcd696d241fc not found: ID does not exist"
Jul 28 17:23:05.602814 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.602762    2573 scope.go:117] "RemoveContainer" containerID="e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"
Jul 28 17:23:05.602982 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:05.602958    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7\": container with ID starting with e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7 not found: ID does not exist" containerID="e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"
Jul 28 17:23:05.603021 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.602992    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7"} err="failed to get container status \"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7\": rpc error: code = NotFound desc = could not find container \"e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7\": container with ID starting with e797c152abb6a82bc90277d87089a5b6c1429afe44780d8f8a9b93c387f788a7 not found: ID does not exist"
Jul 28 17:23:05.606792 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.606769    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:23:05.614629 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:05.614611    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b94d0-predictor-d7574d6d9-nd2n7"]
Jul 28 17:23:07.600653 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:07.600622    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" path="/var/lib/kubelet/pods/5ff3495b-f4af-49c3-be73-58aa5137a2bb/volumes"
Jul 28 17:23:10.534850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:10.534807    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:10.582236 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:10.582213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:10.582719 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:10.582692    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:15.534963 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:15.534916    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:15.535461 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:15.535034    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:23:20.535361 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:20.535312    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:20.583446 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:20.583410    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:25.534865 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:25.534829    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:30.534637 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:30.534602    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:30.583554 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:30.583521    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:32.001719 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.001694    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:23:32.143788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.143696    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle\") pod \"e41d297a-7594-472f-8af9-8404ff789851\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") "
Jul 28 17:23:32.143788 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.143736    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") pod \"e41d297a-7594-472f-8af9-8404ff789851\" (UID: \"e41d297a-7594-472f-8af9-8404ff789851\") "
Jul 28 17:23:32.144075 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.144027    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e41d297a-7594-472f-8af9-8404ff789851" (UID: "e41d297a-7594-472f-8af9-8404ff789851"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:32.145786 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.145757    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e41d297a-7594-472f-8af9-8404ff789851" (UID: "e41d297a-7594-472f-8af9-8404ff789851"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:32.245118 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.245075    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e41d297a-7594-472f-8af9-8404ff789851-openshift-service-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:23:32.245118 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.245107    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41d297a-7594-472f-8af9-8404ff789851-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:23:32.677598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.677568    2573 generic.go:356] "Generic (PLEG): container finished" podID="e41d297a-7594-472f-8af9-8404ff789851" containerID="e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f" exitCode=0
Jul 28 17:23:32.677776 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.677608    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"
Jul 28 17:23:32.677776 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.677613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" event={"ID":"e41d297a-7594-472f-8af9-8404ff789851","Type":"ContainerDied","Data":"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"}
Jul 28 17:23:32.677776 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.677649    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt" event={"ID":"e41d297a-7594-472f-8af9-8404ff789851","Type":"ContainerDied","Data":"58bd6409303dcc397b8570949f6be5800c7ab960b41ab4c243db5c0c263dabe7"}
Jul 28 17:23:32.677776 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.677672    2573 scope.go:117] "RemoveContainer" containerID="e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"
Jul 28 17:23:32.685959 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.685944    2573 scope.go:117] "RemoveContainer" containerID="e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"
Jul 28 17:23:32.686222 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:32.686200    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f\": container with ID starting with e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f not found: ID does not exist" containerID="e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"
Jul 28 17:23:32.686273 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.686230    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f"} err="failed to get container status \"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f\": rpc error: code = NotFound desc = could not find container \"e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f\": container with ID starting with e0d4f40171a4e46858836ea22252d346847b1af52e71575dde5e9df339e3f07f not found: ID does not exist"
Jul 28 17:23:32.700059 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.700030    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:23:32.702375 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:32.702356    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b94d0-647d855485-ztlwt"]
Jul 28 17:23:33.601288 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:33.601245    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41d297a-7594-472f-8af9-8404ff789851" path="/var/lib/kubelet/pods/e41d297a-7594-472f-8af9-8404ff789851/volumes"
Jul 28 17:23:40.582685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:40.582643    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:23:50.583510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:50.583479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:23:51.664711 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.664672    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.664971    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.664981    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.664991    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.664997    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.665008    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kube-rbac-proxy"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.665015    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kube-rbac-proxy"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.665074    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kube-rbac-proxy"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.665085    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41d297a-7594-472f-8af9-8404ff789851" containerName="switch-graph-b94d0"
Jul 28 17:23:51.665096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.665095    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff3495b-f4af-49c3-be73-58aa5137a2bb" containerName="kserve-container"
Jul 28 17:23:51.667668 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.667650    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:51.670560 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.670540    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 17:23:51.670670 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.670577    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 17:23:51.670757 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.670732    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:51.678061 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.678029    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:23:51.686340 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.686315    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:51.686447 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.686357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:51.787424 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.787387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:51.787614 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.787450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:51.787614 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:51.787547    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 28 17:23:51.787614 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:23:51.787610    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls podName:f01437bd-789e-4b2f-a441-7c7fd3c7e0ce nodeName:}" failed. No retries permitted until 2026-07-28 17:23:52.287592736 +0000 UTC m=+875.215514370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls") pod "model-chainer-64698579b9-xlx8n" (UID: "f01437bd-789e-4b2f-a441-7c7fd3c7e0ce") : secret "model-chainer-serving-cert" not found
Jul 28 17:23:51.788131 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:51.788108    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:52.292829 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.292798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:52.295241 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.295221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") pod \"model-chainer-64698579b9-xlx8n\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") " pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:52.577829 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.577743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:52.703776 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.703745    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:23:52.706918 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:23:52.706894    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01437bd_789e_4b2f_a441_7c7fd3c7e0ce.slice/crio-358ea4f4f16aa79220eaa1f312a49bf9f4a6bd6a4d81464c6a98157a2736fa03 WatchSource:0}: Error finding container 358ea4f4f16aa79220eaa1f312a49bf9f4a6bd6a4d81464c6a98157a2736fa03: Status 404 returned error can't find the container with id 358ea4f4f16aa79220eaa1f312a49bf9f4a6bd6a4d81464c6a98157a2736fa03
Jul 28 17:23:52.749825 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.749797    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" event={"ID":"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce","Type":"ContainerStarted","Data":"358ea4f4f16aa79220eaa1f312a49bf9f4a6bd6a4d81464c6a98157a2736fa03"}
Jul 28 17:23:52.876970 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:52.876888    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:53.755445 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:53.755411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" event={"ID":"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce","Type":"ContainerStarted","Data":"b03b4945c0a62e73ef06799eb77a9d54df9b887fa1e689d8a097352b88d5ae56"}
Jul 28 17:23:53.755827 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:53.755520    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:23:53.774305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:53.774256    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podStartSLOduration=2.7742427640000002 podStartE2EDuration="2.774242764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:53.773737569 +0000 UTC m=+876.701659228" watchObservedRunningTime="2026-07-28 17:23:53.774242764 +0000 UTC m=+876.702164449"
Jul 28 17:23:59.763775 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:23:59.763744    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:24:01.773871 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.773838    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:24:01.774257 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.774028    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" containerID="cri-o://b03b4945c0a62e73ef06799eb77a9d54df9b887fa1e689d8a097352b88d5ae56" gracePeriod=30
Jul 28 17:24:01.870574 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.870542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:01.963946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.963911    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:01.982902 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.982873    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:24:01.986365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.986344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:01.989328 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.989308    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-211cd-predictor-serving-cert\""
Jul 28 17:24:01.989328 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.989322    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-211cd-kube-rbac-proxy-sar-config\""
Jul 28 17:24:01.996365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:01.996343    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:24:02.039239 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.039159    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.067630 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.067599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt7c5\" (UniqueName: \"kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.067795 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.067656    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.067795 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.067711    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.133403 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.133370    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:24:02.136023 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.135962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.137287 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.137268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.140355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.140335    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-391b7-kube-rbac-proxy-sar-config\""
Jul 28 17:24:02.140654 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.140638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-391b7-serving-cert\""
Jul 28 17:24:02.150076 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.150041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:24:02.168307 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt7c5\" (UniqueName: \"kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.168307 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168309    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.168460 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.168460 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.168460 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.168582 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:24:02.168512    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-211cd-predictor-serving-cert: secret "success-200-isvc-211cd-predictor-serving-cert" not found
Jul 28 17:24:02.168582 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:24:02.168575    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls podName:37d036fa-be96-4ebe-a842-4df4ebb4355a nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.668554954 +0000 UTC m=+885.596476599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls") pod "success-200-isvc-211cd-predictor-5f9579f849-vpgx8" (UID: "37d036fa-be96-4ebe-a842-4df4ebb4355a") : secret "success-200-isvc-211cd-predictor-serving-cert" not found
Jul 28 17:24:02.168962 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.168945    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.179837 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.179811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt7c5\" (UniqueName: \"kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.208422 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.208394    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.269598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.269559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.269598 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.269604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.269839 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:24:02.269713    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-391b7-serving-cert: secret "switch-graph-391b7-serving-cert" not found
Jul 28 17:24:02.269839 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:24:02.269790    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls podName:1740b307-ce88-4b2b-9ca1-811b2f70144a nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.769773845 +0000 UTC m=+885.697695486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls") pod "switch-graph-391b7-6f8f74c7b5-hsztp" (UID: "1740b307-ce88-4b2b-9ca1-811b2f70144a") : secret "switch-graph-391b7-serving-cert" not found
Jul 28 17:24:02.270265 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.270243    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.314090 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.313987    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.443732 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.443697    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.559157 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.559118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.673490 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.673461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.675807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.675777    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") pod \"success-200-isvc-211cd-predictor-5f9579f849-vpgx8\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") " pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:02.774701 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.774659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.777134 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.777103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") pod \"switch-graph-391b7-6f8f74c7b5-hsztp\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") " pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:02.897012 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:02.896974    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:03.029982 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.029843    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:24:03.032589 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:24:03.032559    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d036fa_be96_4ebe_a842_4df4ebb4355a.slice/crio-eea5e80b5a98015131456e2e7f3b4330ed438e5b2d7fdbeb21f1360326c69f8a WatchSource:0}: Error finding container eea5e80b5a98015131456e2e7f3b4330ed438e5b2d7fdbeb21f1360326c69f8a: Status 404 returned error can't find the container with id eea5e80b5a98015131456e2e7f3b4330ed438e5b2d7fdbeb21f1360326c69f8a
Jul 28 17:24:03.047105 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.046860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:03.180182 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.180157    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:24:03.183477 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:24:03.183449    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1740b307_ce88_4b2b_9ca1_811b2f70144a.slice/crio-8be5316816a698c8d1522795c77d490dc11555725930e3a10ee9ff0436234791 WatchSource:0}: Error finding container 8be5316816a698c8d1522795c77d490dc11555725930e3a10ee9ff0436234791: Status 404 returned error can't find the container with id 8be5316816a698c8d1522795c77d490dc11555725930e3a10ee9ff0436234791
Jul 28 17:24:03.790852 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.790812    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" event={"ID":"1740b307-ce88-4b2b-9ca1-811b2f70144a","Type":"ContainerStarted","Data":"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"}
Jul 28 17:24:03.791305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.790876    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:03.791305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.790897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" event={"ID":"1740b307-ce88-4b2b-9ca1-811b2f70144a","Type":"ContainerStarted","Data":"8be5316816a698c8d1522795c77d490dc11555725930e3a10ee9ff0436234791"}
Jul 28 17:24:03.792497 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.792473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerStarted","Data":"c9ec8e13d0d62834612fa6e22c0d36f84a0fb25985d66076625179eb5a016af3"}
Jul 28 17:24:03.792612 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.792525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerStarted","Data":"a28bb4276e83e9ed0e9e3876379f280bbf9d1b3ea5de02a017c4d89f6176ef43"}
Jul 28 17:24:03.792612 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.792559    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:03.792612 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.792588    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:03.792769 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.792621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerStarted","Data":"eea5e80b5a98015131456e2e7f3b4330ed438e5b2d7fdbeb21f1360326c69f8a"}
Jul 28 17:24:03.794007 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.793983    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:03.812311 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.812269    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podStartSLOduration=1.812256338 podStartE2EDuration="1.812256338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:03.810379312 +0000 UTC m=+886.738300967" watchObservedRunningTime="2026-07-28 17:24:03.812256338 +0000 UTC m=+886.740177992"
Jul 28 17:24:03.830743 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:03.830695    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podStartSLOduration=2.83068116 podStartE2EDuration="2.83068116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:03.829909228 +0000 UTC m=+886.757830884" watchObservedRunningTime="2026-07-28 17:24:03.83068116 +0000 UTC m=+886.758602815"
Jul 28 17:24:04.763030 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:04.762992    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:04.796869 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:04.796826    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:09.762633 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:09.762540    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:09.802014 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:09.801985    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:24:09.802430 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:09.802410    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:24:09.802545 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:09.802485    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:14.763065 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:14.763008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:14.763558 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:14.763132    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:24:19.762487 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:19.762445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:19.802437 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:19.802403    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:24.762566 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:24.762526    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:29.762760 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:29.762713    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:29.803203 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:29.803166    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:31.898813 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:31.898788    2573 generic.go:356] "Generic (PLEG): container finished" podID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerID="b03b4945c0a62e73ef06799eb77a9d54df9b887fa1e689d8a097352b88d5ae56" exitCode=0
Jul 28 17:24:31.899136 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:31.898863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" event={"ID":"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce","Type":"ContainerDied","Data":"b03b4945c0a62e73ef06799eb77a9d54df9b887fa1e689d8a097352b88d5ae56"}
Jul 28 17:24:31.914747 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:31.914727    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:24:32.024840 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.024807    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle\") pod \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") "
Jul 28 17:24:32.025009 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.024874    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") pod \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\" (UID: \"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce\") "
Jul 28 17:24:32.025171 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.025147    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" (UID: "f01437bd-789e-4b2f-a441-7c7fd3c7e0ce"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:32.026856 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.026831    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" (UID: "f01437bd-789e-4b2f-a441-7c7fd3c7e0ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:32.125399 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.125365    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:24:32.125399 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.125395    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce-openshift-service-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:24:32.710070 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.710017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.904241 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.904211    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"
Jul 28 17:24:32.904678 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.904212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n" event={"ID":"f01437bd-789e-4b2f-a441-7c7fd3c7e0ce","Type":"ContainerDied","Data":"358ea4f4f16aa79220eaa1f312a49bf9f4a6bd6a4d81464c6a98157a2736fa03"}
Jul 28 17:24:32.904678 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.904338    2573 scope.go:117] "RemoveContainer" containerID="b03b4945c0a62e73ef06799eb77a9d54df9b887fa1e689d8a097352b88d5ae56"
Jul 28 17:24:32.926286 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.926264    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:24:32.930390 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:32.930373    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-64698579b9-xlx8n"]
Jul 28 17:24:33.605005 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:33.602835    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" path="/var/lib/kubelet/pods/f01437bd-789e-4b2f-a441-7c7fd3c7e0ce/volumes"
Jul 28 17:24:39.031452 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:39.031409    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:39.094374 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:39.094333    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:39.803063 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:39.803007    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:24:49.803184 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:24:49.803157    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:25:02.004472 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.004437    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:25:02.004853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.004725    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer"
Jul 28 17:25:02.004853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.004736    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer"
Jul 28 17:25:02.004853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.004796    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01437bd-789e-4b2f-a441-7c7fd3c7e0ce" containerName="model-chainer"
Jul 28 17:25:02.007593 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.007576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.010286 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.010263    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-211cd-serving-cert\""
Jul 28 17:25:02.010399 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.010320    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-211cd-kube-rbac-proxy-sar-config\""
Jul 28 17:25:02.010603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.010581    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:02.014988 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.014969    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:25:02.060210 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.060183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.060325 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.060227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.161484 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.161449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.161646 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.161514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.161646 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:25:02.161598    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-211cd-serving-cert: secret "sequence-graph-211cd-serving-cert" not found
Jul 28 17:25:02.161721 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:25:02.161676    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls podName:ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:02.66165798 +0000 UTC m=+945.589579614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls") pod "sequence-graph-211cd-755c5c64d7-dsdg5" (UID: "ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4") : secret "sequence-graph-211cd-serving-cert" not found
Jul 28 17:25:02.162108 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.162093    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.597879 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.597845    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:02.666495 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.666456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.669030 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.668999    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") pod \"sequence-graph-211cd-755c5c64d7-dsdg5\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") " pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:02.918883 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:02.918855    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:03.048281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:03.048239    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:25:03.050567 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:25:03.050545    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb64d7b_d69f_4fe6_a4df_a2dc2a97b5a4.slice/crio-932f65b5eb602681a85054e7157c0f9bd3198e95b8d39f6f86db089d54e9a3c1 WatchSource:0}: Error finding container 932f65b5eb602681a85054e7157c0f9bd3198e95b8d39f6f86db089d54e9a3c1: Status 404 returned error can't find the container with id 932f65b5eb602681a85054e7157c0f9bd3198e95b8d39f6f86db089d54e9a3c1
Jul 28 17:25:04.014510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:04.014468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" event={"ID":"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4","Type":"ContainerStarted","Data":"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"}
Jul 28 17:25:04.014681 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:04.014540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" event={"ID":"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4","Type":"ContainerStarted","Data":"932f65b5eb602681a85054e7157c0f9bd3198e95b8d39f6f86db089d54e9a3c1"}
Jul 28 17:25:04.014681 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:04.014568    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:25:04.033121 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:04.033072    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podStartSLOduration=3.033041729 podStartE2EDuration="3.033041729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:04.031770751 +0000 UTC m=+946.959692398" watchObservedRunningTime="2026-07-28 17:25:04.033041729 +0000 UTC m=+946.960963384"
Jul 28 17:25:10.024728 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:25:10.024698    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:26:39.566249 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:26:39.566198    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:39.618958 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:26:39.618924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:38.391863 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:27:38.391830    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:16.858233 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.858195    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:32:16.860685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.858462    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" containerID="cri-o://788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60" gracePeriod=30
Jul 28 17:32:16.903730 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.903700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:16.977526 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.977490    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:32:16.977780 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.977753    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container" containerID="cri-o://85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d" gracePeriod=30
Jul 28 17:32:16.977878 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:16.977805    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kube-rbac-proxy" containerID="cri-o://6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44" gracePeriod=30
Jul 28 17:32:17.045931 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.045902    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:32:17.049088 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.049068    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.052242 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.052224    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a23a-kube-rbac-proxy-sar-config\""
Jul 28 17:32:17.052325 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.052223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a23a-predictor-serving-cert\""
Jul 28 17:32:17.064896 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.064873    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:32:17.081351 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.081317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gdm\" (UniqueName: \"kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.081588 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.081554    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.081686 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.081655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.182937 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.182903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.182937 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.182941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.183193 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.182979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gdm\" (UniqueName: \"kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.183625 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.183603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.185529 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.185507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.196204 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.196181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gdm\" (UniqueName: \"kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm\") pod \"success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") " pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.358759 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.358723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:17.493348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.493324    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:32:17.495718 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:32:17.495688    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4da259a_bf2f_41d2_98c4_1ab1272c65de.slice/crio-f29d4808a3de035f0031efa0ff73c200beefc3d61ae7ac5f6afc7fb485ecad26 WatchSource:0}: Error finding container f29d4808a3de035f0031efa0ff73c200beefc3d61ae7ac5f6afc7fb485ecad26: Status 404 returned error can't find the container with id f29d4808a3de035f0031efa0ff73c200beefc3d61ae7ac5f6afc7fb485ecad26
Jul 28 17:32:17.620782 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.620750    2573 generic.go:356] "Generic (PLEG): container finished" podID="01b01629-5732-466b-809b-4fd5067df411" containerID="6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44" exitCode=2
Jul 28 17:32:17.620952 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.620820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerDied","Data":"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"}
Jul 28 17:32:17.622160 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.622140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerStarted","Data":"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"}
Jul 28 17:32:17.622250 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.622183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerStarted","Data":"f29d4808a3de035f0031efa0ff73c200beefc3d61ae7ac5f6afc7fb485ecad26"}
Jul 28 17:32:17.981212 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:17.981182    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:18.627887 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:18.627851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerStarted","Data":"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"}
Jul 28 17:32:18.628096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:18.628075    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:18.628240 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:18.628223    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:18.629484 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:18.629462    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:18.650603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:18.650547    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podStartSLOduration=1.650527953 podStartE2EDuration="1.650527953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:18.649089781 +0000 UTC m=+1381.577011437" watchObservedRunningTime="2026-07-28 17:32:18.650527953 +0000 UTC m=+1381.578449609"
Jul 28 17:32:19.631457 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:19.631421    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:19.800970 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:19.800930    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:20.217216 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.217193    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:32:20.304150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.304040    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config\") pod \"01b01629-5732-466b-809b-4fd5067df411\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") "
Jul 28 17:32:20.304150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.304118    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmg4m\" (UniqueName: \"kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m\") pod \"01b01629-5732-466b-809b-4fd5067df411\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") "
Jul 28 17:32:20.304150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.304135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") pod \"01b01629-5732-466b-809b-4fd5067df411\" (UID: \"01b01629-5732-466b-809b-4fd5067df411\") "
Jul 28 17:32:20.304502 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.304467    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01b01629-5732-466b-809b-4fd5067df411" (UID: "01b01629-5732-466b-809b-4fd5067df411"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:20.306234 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.306210    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m" (OuterVolumeSpecName: "kube-api-access-zmg4m") pod "01b01629-5732-466b-809b-4fd5067df411" (UID: "01b01629-5732-466b-809b-4fd5067df411"). InnerVolumeSpecName "kube-api-access-zmg4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:20.306234 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.306212    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01b01629-5732-466b-809b-4fd5067df411" (UID: "01b01629-5732-466b-809b-4fd5067df411"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:20.404950 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.404899    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmg4m\" (UniqueName: \"kubernetes.io/projected/01b01629-5732-466b-809b-4fd5067df411-kube-api-access-zmg4m\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:32:20.404950 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.404941    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b01629-5732-466b-809b-4fd5067df411-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:32:20.404950 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.404951    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b01629-5732-466b-809b-4fd5067df411-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:32:20.635987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.635903    2573 generic.go:356] "Generic (PLEG): container finished" podID="01b01629-5732-466b-809b-4fd5067df411" containerID="85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d" exitCode=0
Jul 28 17:32:20.635987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.635954    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"
Jul 28 17:32:20.635987 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.635975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerDied","Data":"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"}
Jul 28 17:32:20.636480 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.636024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r" event={"ID":"01b01629-5732-466b-809b-4fd5067df411","Type":"ContainerDied","Data":"e08ff005f24f272559c23394639138731811e7923166b276f819ce10a0cb66b0"}
Jul 28 17:32:20.636480 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.636064    2573 scope.go:117] "RemoveContainer" containerID="6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"
Jul 28 17:32:20.645001 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.644982    2573 scope.go:117] "RemoveContainer" containerID="85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"
Jul 28 17:32:20.652565 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.652543    2573 scope.go:117] "RemoveContainer" containerID="6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"
Jul 28 17:32:20.652881 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:32:20.652852    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44\": container with ID starting with 6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44 not found: ID does not exist" containerID="6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"
Jul 28 17:32:20.652980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.652892    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44"} err="failed to get container status \"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44\": rpc error: code = NotFound desc = could not find container \"6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44\": container with ID starting with 6040dc43e83b1c2b3a9f636b7d32a31b27d965c4a9b9dc6b80c765790abc2d44 not found: ID does not exist"
Jul 28 17:32:20.652980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.652919    2573 scope.go:117] "RemoveContainer" containerID="85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"
Jul 28 17:32:20.653151 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:32:20.653132    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d\": container with ID starting with 85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d not found: ID does not exist" containerID="85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"
Jul 28 17:32:20.653203 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.653157    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d"} err="failed to get container status \"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d\": rpc error: code = NotFound desc = could not find container \"85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d\": container with ID starting with 85e6574e28c6421be7e4d2092c2faf98a7755cc5a6fb6e8b32bb82b9b291db8d not found: ID does not exist"
Jul 28 17:32:20.658889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.658869    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:32:20.663454 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:20.663415    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-391b7-predictor-888b7969f-kw99r"]
Jul 28 17:32:21.601346 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:21.601311    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b01629-5732-466b-809b-4fd5067df411" path="/var/lib/kubelet/pods/01b01629-5732-466b-809b-4fd5067df411/volumes"
Jul 28 17:32:24.635965 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:24.635933    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:32:24.636438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:24.636395    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:24.800817 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:24.800780    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:29.800762 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:29.800715    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:29.801161 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:29.800838    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:32:34.636582 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:34.636535    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:34.800928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:34.800885    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:39.800904 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:39.800866    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:44.637315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:44.637271    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:44.801026 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:44.800986    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:47.011107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.011080    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:32:47.120467 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.120367    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") pod \"1740b307-ce88-4b2b-9ca1-811b2f70144a\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") "
Jul 28 17:32:47.120641 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.120471    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle\") pod \"1740b307-ce88-4b2b-9ca1-811b2f70144a\" (UID: \"1740b307-ce88-4b2b-9ca1-811b2f70144a\") "
Jul 28 17:32:47.120824 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.120795    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1740b307-ce88-4b2b-9ca1-811b2f70144a" (UID: "1740b307-ce88-4b2b-9ca1-811b2f70144a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:47.122433 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.122408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1740b307-ce88-4b2b-9ca1-811b2f70144a" (UID: "1740b307-ce88-4b2b-9ca1-811b2f70144a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:47.221661 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.221623    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1740b307-ce88-4b2b-9ca1-811b2f70144a-openshift-service-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:32:47.221661 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.221649    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1740b307-ce88-4b2b-9ca1-811b2f70144a-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:32:47.737197 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.737161    2573 generic.go:356] "Generic (PLEG): container finished" podID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerID="788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60" exitCode=0
Jul 28 17:32:47.737366 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.737222    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"
Jul 28 17:32:47.737366 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.737249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" event={"ID":"1740b307-ce88-4b2b-9ca1-811b2f70144a","Type":"ContainerDied","Data":"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"}
Jul 28 17:32:47.737366 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.737309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp" event={"ID":"1740b307-ce88-4b2b-9ca1-811b2f70144a","Type":"ContainerDied","Data":"8be5316816a698c8d1522795c77d490dc11555725930e3a10ee9ff0436234791"}
Jul 28 17:32:47.737366 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.737336    2573 scope.go:117] "RemoveContainer" containerID="788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"
Jul 28 17:32:47.745717 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.745697    2573 scope.go:117] "RemoveContainer" containerID="788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"
Jul 28 17:32:47.745968 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:32:47.745947    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60\": container with ID starting with 788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60 not found: ID does not exist" containerID="788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"
Jul 28 17:32:47.746043 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.745979    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60"} err="failed to get container status \"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60\": rpc error: code = NotFound desc = could not find container \"788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60\": container with ID starting with 788f6ccc96c648ab2dc94354de10f4f23186b155c5053bc795521dcbb3376e60 not found: ID does not exist"
Jul 28 17:32:47.757407 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.757385    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:32:47.761384 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:47.761364    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-391b7-6f8f74c7b5-hsztp"]
Jul 28 17:32:49.600405 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:49.600365    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" path="/var/lib/kubelet/pods/1740b307-ce88-4b2b-9ca1-811b2f70144a/volumes"
Jul 28 17:32:54.636545 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:32:54.636500    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:33:04.637621 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:04.637586    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:33:16.680914 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.680825    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:33:16.681438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.681042    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" containerID="cri-o://f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416" gracePeriod=30
Jul 28 17:33:16.799017 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.798974    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:33:16.799244 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.799215    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" containerID="cri-o://a28bb4276e83e9ed0e9e3876379f280bbf9d1b3ea5de02a017c4d89f6176ef43" gracePeriod=30
Jul 28 17:33:16.799333 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.799248    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kube-rbac-proxy" containerID="cri-o://c9ec8e13d0d62834612fa6e22c0d36f84a0fb25985d66076625179eb5a016af3" gracePeriod=30
Jul 28 17:33:16.925465 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925430    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:33:16.925841 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925823    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container"
Jul 28 17:33:16.925930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925844    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container"
Jul 28 17:33:16.925930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925861    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kube-rbac-proxy"
Jul 28 17:33:16.925930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925870    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kube-rbac-proxy"
Jul 28 17:33:16.925930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7"
Jul 28 17:33:16.925930 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7"
Jul 28 17:33:16.926207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925968    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kube-rbac-proxy"
Jul 28 17:33:16.926207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925983    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1740b307-ce88-4b2b-9ca1-811b2f70144a" containerName="switch-graph-391b7"
Jul 28 17:33:16.926207 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.925993    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b01629-5732-466b-809b-4fd5067df411" containerName="kserve-container"
Jul 28 17:33:16.929140 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.929117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:16.932201 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.932130    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ef873-predictor-serving-cert\""
Jul 28 17:33:16.932335 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.932196    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ef873-kube-rbac-proxy-sar-config\""
Jul 28 17:33:16.938935 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.938906    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:33:16.964714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.964668    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:16.964714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.964710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:16.964924 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:16.964752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskm8\" (UniqueName: \"kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.065442 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.065393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.065442 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.065443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.065801 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.065512    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskm8\" (UniqueName: \"kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.065801 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:17.065629    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ef873-predictor-serving-cert: secret "success-200-isvc-ef873-predictor-serving-cert" not found
Jul 28 17:33:17.065801 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:17.065701    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls podName:0715e64e-bb8d-4c48-b07d-724573ab36c0 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:17.565679134 +0000 UTC m=+1440.493600771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls") pod "success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" (UID: "0715e64e-bb8d-4c48-b07d-724573ab36c0") : secret "success-200-isvc-ef873-predictor-serving-cert" not found
Jul 28 17:33:17.066245 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.066213    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.083853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.083820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskm8\" (UniqueName: \"kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.569922 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.569902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.572355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.572333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") pod \"success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") " pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.840966 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.840874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:17.852428 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.852395    2573 generic.go:356] "Generic (PLEG): container finished" podID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerID="c9ec8e13d0d62834612fa6e22c0d36f84a0fb25985d66076625179eb5a016af3" exitCode=2
Jul 28 17:33:17.852530 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.852473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerDied","Data":"c9ec8e13d0d62834612fa6e22c0d36f84a0fb25985d66076625179eb5a016af3"}
Jul 28 17:33:17.977946 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:17.977910    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:33:17.980696 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:33:17.980668    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0715e64e_bb8d_4c48_b07d_724573ab36c0.slice/crio-b9d891beebff6fa9151217aaddd02aac499074f5e330e0417ccb8f60524fffcd WatchSource:0}: Error finding container b9d891beebff6fa9151217aaddd02aac499074f5e330e0417ccb8f60524fffcd: Status 404 returned error can't find the container with id b9d891beebff6fa9151217aaddd02aac499074f5e330e0417ccb8f60524fffcd
Jul 28 17:33:18.251337 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.251303    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:18.352281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.352203    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:18.857610 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.857576    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerStarted","Data":"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"}
Jul 28 17:33:18.858044 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.857634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerStarted","Data":"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"}
Jul 28 17:33:18.858044 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.857653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerStarted","Data":"b9d891beebff6fa9151217aaddd02aac499074f5e330e0417ccb8f60524fffcd"}
Jul 28 17:33:18.858044 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.857866    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:18.858044 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.857889    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:18.859527 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.859499    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:18.879303 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:18.879252    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podStartSLOduration=2.879238595 podStartE2EDuration="2.879238595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:18.876780006 +0000 UTC m=+1441.804701671" watchObservedRunningTime="2026-07-28 17:33:18.879238595 +0000 UTC m=+1441.807160291"
Jul 28 17:33:19.797393 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.797360    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 17:33:19.802807 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.802780    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:33:19.866359 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.866331    2573 generic.go:356] "Generic (PLEG): container finished" podID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerID="a28bb4276e83e9ed0e9e3876379f280bbf9d1b3ea5de02a017c4d89f6176ef43" exitCode=0
Jul 28 17:33:19.866712 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.866409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerDied","Data":"a28bb4276e83e9ed0e9e3876379f280bbf9d1b3ea5de02a017c4d89f6176ef43"}
Jul 28 17:33:19.866826 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.866802    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:19.957724 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.957698    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:33:19.989802 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.989774    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config\") pod \"37d036fa-be96-4ebe-a842-4df4ebb4355a\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") "
Jul 28 17:33:19.989802 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.989806    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") pod \"37d036fa-be96-4ebe-a842-4df4ebb4355a\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") "
Jul 28 17:33:19.989989 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.989838    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt7c5\" (UniqueName: \"kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5\") pod \"37d036fa-be96-4ebe-a842-4df4ebb4355a\" (UID: \"37d036fa-be96-4ebe-a842-4df4ebb4355a\") "
Jul 28 17:33:19.990169 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.990143    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37d036fa-be96-4ebe-a842-4df4ebb4355a" (UID: "37d036fa-be96-4ebe-a842-4df4ebb4355a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:19.991937 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.991906    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37d036fa-be96-4ebe-a842-4df4ebb4355a" (UID: "37d036fa-be96-4ebe-a842-4df4ebb4355a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:19.991937 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:19.991928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5" (OuterVolumeSpecName: "kube-api-access-qt7c5") pod "37d036fa-be96-4ebe-a842-4df4ebb4355a" (UID: "37d036fa-be96-4ebe-a842-4df4ebb4355a"). InnerVolumeSpecName "kube-api-access-qt7c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:20.023663 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.023604    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:20.090354 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.090315    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:20.090354 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.090353    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d036fa-be96-4ebe-a842-4df4ebb4355a-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:20.090354 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.090364    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt7c5\" (UniqueName: \"kubernetes.io/projected/37d036fa-be96-4ebe-a842-4df4ebb4355a-kube-api-access-qt7c5\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:20.871649 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.871617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8" event={"ID":"37d036fa-be96-4ebe-a842-4df4ebb4355a","Type":"ContainerDied","Data":"eea5e80b5a98015131456e2e7f3b4330ed438e5b2d7fdbeb21f1360326c69f8a"}
Jul 28 17:33:20.871649 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.871642    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"
Jul 28 17:33:20.872224 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.871680    2573 scope.go:117] "RemoveContainer" containerID="c9ec8e13d0d62834612fa6e22c0d36f84a0fb25985d66076625179eb5a016af3"
Jul 28 17:33:20.879773 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.879756    2573 scope.go:117] "RemoveContainer" containerID="a28bb4276e83e9ed0e9e3876379f280bbf9d1b3ea5de02a017c4d89f6176ef43"
Jul 28 17:33:20.893897 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.893874    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:33:20.899733 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:20.899711    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-211cd-predictor-5f9579f849-vpgx8"]
Jul 28 17:33:21.600444 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:21.600412    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" path="/var/lib/kubelet/pods/37d036fa-be96-4ebe-a842-4df4ebb4355a/volumes"
Jul 28 17:33:24.871777 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:24.871744    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:33:24.872305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:24.872274    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:25.022693 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:25.022657    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:30.023856 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:30.023817    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:30.024255 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:30.023928    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:33:31.245102 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.245069    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.323099 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.323064    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.349356 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.349325    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:33:31.350251 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.350206    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kube-rbac-proxy" containerID="cri-o://18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b" gracePeriod=30
Jul 28 17:33:31.350599 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.350579    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" containerID="cri-o://57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316" gracePeriod=30
Jul 28 17:33:31.399151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.399075    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.400522 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400500    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:33:31.400798 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400787    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container"
Jul 28 17:33:31.400850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400800    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container"
Jul 28 17:33:31.400850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400809    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kube-rbac-proxy"
Jul 28 17:33:31.400850 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400814    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kube-rbac-proxy"
Jul 28 17:33:31.400957 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400860    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kube-rbac-proxy"
Jul 28 17:33:31.400957 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.400869    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d036fa-be96-4ebe-a842-4df4ebb4355a" containerName="kserve-container"
Jul 28 17:33:31.405315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.405297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.412570 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.412531    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1fc0-kube-rbac-proxy-sar-config\""
Jul 28 17:33:31.412820 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.412801    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d1fc0-predictor-serving-cert\""
Jul 28 17:33:31.415339 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.415312    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:33:31.488820 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.488789    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.488981 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.488840    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.488981 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.488865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsdh\" (UniqueName: \"kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.495748 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.495683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.561611 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.561581    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.589540 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.589502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.589714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.589551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsdh\" (UniqueName: \"kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.589714 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.589627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.589820 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:31.589740    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-serving-cert: secret "success-200-isvc-d1fc0-predictor-serving-cert" not found
Jul 28 17:33:31.589820 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:31.589816    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls podName:8516fcf2-f6ee-4544-9dd7-56ccda7afae2 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:32.089792836 +0000 UTC m=+1455.017714491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls") pod "success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" (UID: "8516fcf2-f6ee-4544-9dd7-56ccda7afae2") : secret "success-200-isvc-d1fc0-predictor-serving-cert" not found
Jul 28 17:33:31.590280 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.590254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.603858 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.603827    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsdh\" (UniqueName: \"kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:31.920823 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.920794    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerID="18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b" exitCode=2
Jul 28 17:33:31.920993 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.920848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerDied","Data":"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"}
Jul 28 17:33:31.963289 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:31.963249    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:32.092897 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.092857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:32.095243 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.095221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") pod \"success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") " pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:32.321399 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.321310    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:32.449806 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.449776    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:33:32.451965 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:33:32.451933    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8516fcf2_f6ee_4544_9dd7_56ccda7afae2.slice/crio-72ac15b7b6ec4d7f3e933f72ca25492bd1f19896cbf7c0eb1e42354b965b3530 WatchSource:0}: Error finding container 72ac15b7b6ec4d7f3e933f72ca25492bd1f19896cbf7c0eb1e42354b965b3530: Status 404 returned error can't find the container with id 72ac15b7b6ec4d7f3e933f72ca25492bd1f19896cbf7c0eb1e42354b965b3530
Jul 28 17:33:32.926420 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.926379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerStarted","Data":"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"}
Jul 28 17:33:32.926585 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.926456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerStarted","Data":"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"}
Jul 28 17:33:32.926585 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.926487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerStarted","Data":"72ac15b7b6ec4d7f3e933f72ca25492bd1f19896cbf7c0eb1e42354b965b3530"}
Jul 28 17:33:32.926585 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.926520    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:32.945548 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:32.945488    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podStartSLOduration=1.945467826 podStartE2EDuration="1.945467826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:32.944384161 +0000 UTC m=+1455.872305831" watchObservedRunningTime="2026-07-28 17:33:32.945467826 +0000 UTC m=+1455.873389482"
Jul 28 17:33:33.930369 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:33.930342    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:33.931515 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:33.931482    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:34.632477 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.632436    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 17:33:34.636724 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.636698    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:33:34.872603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.872566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:34.902677 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.902641    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:33:34.936367 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936334    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerID="57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316" exitCode=0
Jul 28 17:33:34.936772 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936387    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"
Jul 28 17:33:34.936772 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerDied","Data":"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"}
Jul 28 17:33:34.936772 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz" event={"ID":"e4da259a-bf2f-41d2-98c4-1ab1272c65de","Type":"ContainerDied","Data":"f29d4808a3de035f0031efa0ff73c200beefc3d61ae7ac5f6afc7fb485ecad26"}
Jul 28 17:33:34.936772 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936514    2573 scope.go:117] "RemoveContainer" containerID="18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"
Jul 28 17:33:34.936957 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.936829    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:34.945700 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.945679    2573 scope.go:117] "RemoveContainer" containerID="57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"
Jul 28 17:33:34.954927 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.954899    2573 scope.go:117] "RemoveContainer" containerID="18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"
Jul 28 17:33:34.956974 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:34.956948    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b\": container with ID starting with 18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b not found: ID does not exist" containerID="18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"
Jul 28 17:33:34.957117 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.956982    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b"} err="failed to get container status \"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b\": rpc error: code = NotFound desc = could not find container \"18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b\": container with ID starting with 18bd3d6cb5c84192e45ccc80bf28a11fa91fab37f2fbd01ca3f95b912394947b not found: ID does not exist"
Jul 28 17:33:34.957117 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.957001    2573 scope.go:117] "RemoveContainer" containerID="57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"
Jul 28 17:33:34.957311 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:34.957295    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316\": container with ID starting with 57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316 not found: ID does not exist" containerID="57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"
Jul 28 17:33:34.957372 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:34.957316    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316"} err="failed to get container status \"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316\": rpc error: code = NotFound desc = could not find container \"57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316\": container with ID starting with 57e5435b2712bf19833a4620b047e50f1ae3a83176d4affca64c254334878316 not found: ID does not exist"
Jul 28 17:33:35.017297 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.017259    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config\") pod \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") "
Jul 28 17:33:35.017297 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.017303    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls\") pod \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") "
Jul 28 17:33:35.017498 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.017351    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64gdm\" (UniqueName: \"kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm\") pod \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\" (UID: \"e4da259a-bf2f-41d2-98c4-1ab1272c65de\") "
Jul 28 17:33:35.017712 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.017687    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4da259a-bf2f-41d2-98c4-1ab1272c65de" (UID: "e4da259a-bf2f-41d2-98c4-1ab1272c65de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:35.019456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.019434    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4da259a-bf2f-41d2-98c4-1ab1272c65de" (UID: "e4da259a-bf2f-41d2-98c4-1ab1272c65de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:35.019590 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.019565    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm" (OuterVolumeSpecName: "kube-api-access-64gdm") pod "e4da259a-bf2f-41d2-98c4-1ab1272c65de" (UID: "e4da259a-bf2f-41d2-98c4-1ab1272c65de"). InnerVolumeSpecName "kube-api-access-64gdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:35.023089 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.023058    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:35.118359 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.118321    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:35.118359 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.118351    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4da259a-bf2f-41d2-98c4-1ab1272c65de-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:35.118359 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.118362    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gdm\" (UniqueName: \"kubernetes.io/projected/e4da259a-bf2f-41d2-98c4-1ab1272c65de-kube-api-access-64gdm\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:35.271000 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.270967    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:33:35.275742 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.275714    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a23a-predictor-64fbf5fcd5-d5bxz"]
Jul 28 17:33:35.605694 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:35.605618    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" path="/var/lib/kubelet/pods/e4da259a-bf2f-41d2-98c4-1ab1272c65de/volumes"
Jul 28 17:33:39.942668 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:39.942641    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:33:39.943157 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:39.943107    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:40.022853 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:40.022817    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:44.872889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:44.872841    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:45.023034 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:45.022993    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:46.829680 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.829649    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:33:46.921313 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.921263    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle\") pod \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") "
Jul 28 17:33:46.921518 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.921375    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") pod \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\" (UID: \"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4\") "
Jul 28 17:33:46.921643 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.921618    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" (UID: "ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:46.923377 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.923353    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" (UID: "ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:46.985321 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.985231    2573 generic.go:356] "Generic (PLEG): container finished" podID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerID="f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416" exitCode=0
Jul 28 17:33:46.985321 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.985276    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"
Jul 28 17:33:46.985321 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.985299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" event={"ID":"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4","Type":"ContainerDied","Data":"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"}
Jul 28 17:33:46.985596 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.985350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5" event={"ID":"ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4","Type":"ContainerDied","Data":"932f65b5eb602681a85054e7157c0f9bd3198e95b8d39f6f86db089d54e9a3c1"}
Jul 28 17:33:46.985596 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.985376    2573 scope.go:117] "RemoveContainer" containerID="f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"
Jul 28 17:33:46.993281 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.993262    2573 scope.go:117] "RemoveContainer" containerID="f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"
Jul 28 17:33:46.993523 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:33:46.993502    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416\": container with ID starting with f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416 not found: ID does not exist" containerID="f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"
Jul 28 17:33:46.993574 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:46.993531    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416"} err="failed to get container status \"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416\": rpc error: code = NotFound desc = could not find container \"f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416\": container with ID starting with f9bc3b5999455d4549232cf971152fb282f4510a233ae8fa4df5bab60cb79416 not found: ID does not exist"
Jul 28 17:33:47.009918 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:47.009890    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:33:47.016403 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:47.016383    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-211cd-755c5c64d7-dsdg5"]
Jul 28 17:33:47.022375 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:47.022352    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-openshift-service-ca-bundle\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:47.022444 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:47.022376    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:33:47.600560 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:47.600524    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" path="/var/lib/kubelet/pods/ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4/volumes"
Jul 28 17:33:49.943329 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:49.943290    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:54.873107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:54.873032    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:59.943442 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:33:59.943404    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:34:04.873688 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:04.873657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:34:09.943815 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:09.943776    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:34:12.952328 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:12.952296    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:16.959035 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:16.959003    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:17.346913 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:17.346824    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:19.944242 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:19.944208    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:34:27.082491 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.082452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.218247 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.218189    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.250655 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.250617    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:34:27.251185 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.250939    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container" containerID="cri-o://ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c" gracePeriod=30
Jul 28 17:34:27.251185 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.250993    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kube-rbac-proxy" containerID="cri-o://3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a" gracePeriod=30
Jul 28 17:34:27.260404 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260382    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:34:27.260696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260684    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container"
Jul 28 17:34:27.260696 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260697    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260708    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260717    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260727    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kube-rbac-proxy"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260733    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kube-rbac-proxy"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260783    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kube-rbac-proxy"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260790    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4da259a-bf2f-41d2-98c4-1ab1272c65de" containerName="kserve-container"
Jul 28 17:34:27.260808 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.260796    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb64d7b-d69f-4fe6-a4df-a2dc2a97b5a4" containerName="sequence-graph-211cd"
Jul 28 17:34:27.265394 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.265375    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.268300 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.268280    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a513a-predictor-serving-cert\""
Jul 28 17:34:27.268440 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.268418    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a513a-kube-rbac-proxy-sar-config\""
Jul 28 17:34:27.274459 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.274433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:34:27.297716 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.297679    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.353523 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.353494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.353683 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.353587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6qw\" (UniqueName: \"kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.353683 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.353641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.365745 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.365718    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.459673 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.459636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6qw\" (UniqueName: \"kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.459886 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.459867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.459959 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.459904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.460767 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.460736    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.463149 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.463117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.466369 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.466317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.469692 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.469641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6qw\" (UniqueName: \"kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw\") pod \"success-200-isvc-a513a-predictor-5477b77665-h87n4\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") " pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.576911 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.576859    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:27.649247 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.649219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.712540 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:27.712511    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:34:27.715025 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:34:27.714999    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dadb749_2e06_4e5c_9cc3_ac42790048f3.slice/crio-d31de408af355ee3f49d99983c29d01add1fd42dfe8d72a348dc51b01cdb3539 WatchSource:0}: Error finding container d31de408af355ee3f49d99983c29d01add1fd42dfe8d72a348dc51b01cdb3539: Status 404 returned error can't find the container with id d31de408af355ee3f49d99983c29d01add1fd42dfe8d72a348dc51b01cdb3539
Jul 28 17:34:28.141393 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.141282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerStarted","Data":"bdc9d41332a0b09f0fdf85b99d19a6258cac8928f880b4bab723aec6c253714c"}
Jul 28 17:34:28.141393 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.141357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerStarted","Data":"8518573d26344edb6b969ad75806979da1c99e0bfa1364ea9de7c205da73bac9"}
Jul 28 17:34:28.141393 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.141389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerStarted","Data":"d31de408af355ee3f49d99983c29d01add1fd42dfe8d72a348dc51b01cdb3539"}
Jul 28 17:34:28.141951 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.141422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:28.143158 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.143124    2573 generic.go:356] "Generic (PLEG): container finished" podID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerID="3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a" exitCode=2
Jul 28 17:34:28.143294 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.143175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerDied","Data":"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"}
Jul 28 17:34:28.162576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:28.162517    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podStartSLOduration=1.16249557 podStartE2EDuration="1.16249557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:28.161440507 +0000 UTC m=+1511.089362175" watchObservedRunningTime="2026-07-28 17:34:28.16249557 +0000 UTC m=+1511.090417231"
Jul 28 17:34:29.147979 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:29.147948    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:29.149639 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:29.149614    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:29.867438 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:29.867391    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 17:34:30.151457 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.151364    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:30.427996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.427960    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:30.500979 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.500953    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:34:30.587187 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.587153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config\") pod \"0715e64e-bb8d-4c48-b07d-724573ab36c0\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") "
Jul 28 17:34:30.587371 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.587226    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") pod \"0715e64e-bb8d-4c48-b07d-724573ab36c0\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") "
Jul 28 17:34:30.587371 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.587284    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskm8\" (UniqueName: \"kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8\") pod \"0715e64e-bb8d-4c48-b07d-724573ab36c0\" (UID: \"0715e64e-bb8d-4c48-b07d-724573ab36c0\") "
Jul 28 17:34:30.587614 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.587584    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0715e64e-bb8d-4c48-b07d-724573ab36c0" (UID: "0715e64e-bb8d-4c48-b07d-724573ab36c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:30.589341 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.589318    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0715e64e-bb8d-4c48-b07d-724573ab36c0" (UID: "0715e64e-bb8d-4c48-b07d-724573ab36c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:30.589428 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.589377    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8" (OuterVolumeSpecName: "kube-api-access-jskm8") pod "0715e64e-bb8d-4c48-b07d-724573ab36c0" (UID: "0715e64e-bb8d-4c48-b07d-724573ab36c0"). InnerVolumeSpecName "kube-api-access-jskm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:30.688114 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.688017    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0715e64e-bb8d-4c48-b07d-724573ab36c0-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:34:30.688114 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.688076    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskm8\" (UniqueName: \"kubernetes.io/projected/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-api-access-jskm8\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:34:30.688114 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:30.688086    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0715e64e-bb8d-4c48-b07d-724573ab36c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:34:31.157154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.157039    2573 generic.go:356] "Generic (PLEG): container finished" podID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerID="ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c" exitCode=0
Jul 28 17:34:31.157154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.157076    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerDied","Data":"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"}
Jul 28 17:34:31.157154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.157116    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"
Jul 28 17:34:31.157154 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.157140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4" event={"ID":"0715e64e-bb8d-4c48-b07d-724573ab36c0","Type":"ContainerDied","Data":"b9d891beebff6fa9151217aaddd02aac499074f5e330e0417ccb8f60524fffcd"}
Jul 28 17:34:31.157664 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.157168    2573 scope.go:117] "RemoveContainer" containerID="3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"
Jul 28 17:34:31.165439 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.165414    2573 scope.go:117] "RemoveContainer" containerID="ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"
Jul 28 17:34:31.172448 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.172429    2573 scope.go:117] "RemoveContainer" containerID="3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"
Jul 28 17:34:31.172702 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:34:31.172682    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a\": container with ID starting with 3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a not found: ID does not exist" containerID="3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"
Jul 28 17:34:31.172754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.172713    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a"} err="failed to get container status \"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a\": rpc error: code = NotFound desc = could not find container \"3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a\": container with ID starting with 3469a4df65227792c568e067fa8a77deec7bc31fef603ec451eea0823d0f468a not found: ID does not exist"
Jul 28 17:34:31.172754 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.172731    2573 scope.go:117] "RemoveContainer" containerID="ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"
Jul 28 17:34:31.172942 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:34:31.172927    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c\": container with ID starting with ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c not found: ID does not exist" containerID="ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"
Jul 28 17:34:31.172985 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.172948    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c"} err="failed to get container status \"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c\": rpc error: code = NotFound desc = could not find container \"ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c\": container with ID starting with ae04d5e945338cbdb9601f0841f367de520fb69393266727fb07457a41a5674c not found: ID does not exist"
Jul 28 17:34:31.179988 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.179966    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:34:31.183732 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.183710    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ef873-predictor-69fd7d9b9-zhbb4"]
Jul 28 17:34:31.470497 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.470462    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:31.534860 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.534822    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:31.605753 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:31.605705    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" path="/var/lib/kubelet/pods/0715e64e-bb8d-4c48-b07d-724573ab36c0/volumes"
Jul 28 17:34:35.155667 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:35.155636    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:34:35.156213 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:35.156182    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:45.156725 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:45.156632    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:55.157121 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:34:55.157083    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:35:05.156723 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:35:05.156683    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:35:15.157251 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:35:15.157219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:35:27.337243 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:35:27.337212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:27.455259 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:35:27.455227    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:33.988323 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:36:33.988285    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:24.822767 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:41:24.822730    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.083876 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.083842    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.184592 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.184554    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:42:46.184859 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.184830    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container" containerID="cri-o://190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32" gracePeriod=30
Jul 28 17:42:46.185073 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.185023    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kube-rbac-proxy" containerID="cri-o://2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af" gracePeriod=30
Jul 28 17:42:46.200096 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.200075    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.253058 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253015    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:42:46.253319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253307    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container"
Jul 28 17:42:46.253365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253320    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container"
Jul 28 17:42:46.253365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253330    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kube-rbac-proxy"
Jul 28 17:42:46.253365 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253336    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kube-rbac-proxy"
Jul 28 17:42:46.253456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253388    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kserve-container"
Jul 28 17:42:46.253456 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.253398    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0715e64e-bb8d-4c48-b07d-724573ab36c0" containerName="kube-rbac-proxy"
Jul 28 17:42:46.256386 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.256360    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.259404 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.259383    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11994-predictor-serving-cert\""
Jul 28 17:42:46.259569 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.259426    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11994-kube-rbac-proxy-sar-config\""
Jul 28 17:42:46.268885 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.268851    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:42:46.274353 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.274324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.343839 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.343760    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.382057 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.382019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.382221 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.382115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlbh4\" (UniqueName: \"kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.382221 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.382161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.483240 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.483213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlbh4\" (UniqueName: \"kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.483433 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.483254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.483433 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.483302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.484016 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.483996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.485795 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.485773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.490395 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.490369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.493336 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.493317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlbh4\" (UniqueName: \"kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4\") pod \"success-200-isvc-11994-predictor-5bbcdf967f-9qn7z\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") " pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.570570 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.570551    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.699342 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.699318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:42:46.701242 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:42:46.701215    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b0b08b_2f90_49b6_9c54_f2cfa08459fe.slice/crio-603b4f840d7b45d6589b1919524774bbb394381c0d5677b47689894510e8a833 WatchSource:0}: Error finding container 603b4f840d7b45d6589b1919524774bbb394381c0d5677b47689894510e8a833: Status 404 returned error can't find the container with id 603b4f840d7b45d6589b1919524774bbb394381c0d5677b47689894510e8a833
Jul 28 17:42:46.822551 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.822523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.895836 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.895756    2573 generic.go:356] "Generic (PLEG): container finished" podID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerID="2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af" exitCode=2
Jul 28 17:42:46.895992 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.895832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerDied","Data":"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"}
Jul 28 17:42:46.897422 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.897397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerStarted","Data":"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"}
Jul 28 17:42:46.897537 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.897444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerStarted","Data":"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"}
Jul 28 17:42:46.897537 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.897465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerStarted","Data":"603b4f840d7b45d6589b1919524774bbb394381c0d5677b47689894510e8a833"}
Jul 28 17:42:46.897537 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.897512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:46.918403 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:46.918358    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podStartSLOduration=0.918344098 podStartE2EDuration="918.344098ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:46.916641523 +0000 UTC m=+2009.844563178" watchObservedRunningTime="2026-07-28 17:42:46.918344098 +0000 UTC m=+2009.846265753"
Jul 28 17:42:47.900877 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:47.900844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:47.902259 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:47.902226    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:42:48.907063 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:48.907011    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:42:49.136396 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.136373    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:42:49.306708 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.306665    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") pod \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") "
Jul 28 17:42:49.306861 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.306728    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgsdh\" (UniqueName: \"kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh\") pod \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") "
Jul 28 17:42:49.306861 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.306836    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config\") pod \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\" (UID: \"8516fcf2-f6ee-4544-9dd7-56ccda7afae2\") "
Jul 28 17:42:49.307184 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.307155    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8516fcf2-f6ee-4544-9dd7-56ccda7afae2" (UID: "8516fcf2-f6ee-4544-9dd7-56ccda7afae2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:49.308760 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.308736    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8516fcf2-f6ee-4544-9dd7-56ccda7afae2" (UID: "8516fcf2-f6ee-4544-9dd7-56ccda7afae2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:49.308839 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.308814    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh" (OuterVolumeSpecName: "kube-api-access-sgsdh") pod "8516fcf2-f6ee-4544-9dd7-56ccda7afae2" (UID: "8516fcf2-f6ee-4544-9dd7-56ccda7afae2"). InnerVolumeSpecName "kube-api-access-sgsdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:49.407542 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.407519    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:42:49.407542 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.407540    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsdh\" (UniqueName: \"kubernetes.io/projected/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-api-access-sgsdh\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:42:49.407664 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.407549    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8516fcf2-f6ee-4544-9dd7-56ccda7afae2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:42:49.911563 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.911534    2573 generic.go:356] "Generic (PLEG): container finished" podID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerID="190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32" exitCode=0
Jul 28 17:42:49.911954 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.911616    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"
Jul 28 17:42:49.911954 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.911623    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerDied","Data":"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"}
Jul 28 17:42:49.911954 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.911749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd" event={"ID":"8516fcf2-f6ee-4544-9dd7-56ccda7afae2","Type":"ContainerDied","Data":"72ac15b7b6ec4d7f3e933f72ca25492bd1f19896cbf7c0eb1e42354b965b3530"}
Jul 28 17:42:49.911954 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.911776    2573 scope.go:117] "RemoveContainer" containerID="2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"
Jul 28 17:42:49.919305 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.919288    2573 scope.go:117] "RemoveContainer" containerID="190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"
Jul 28 17:42:49.925894 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.925875    2573 scope.go:117] "RemoveContainer" containerID="2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"
Jul 28 17:42:49.926154 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:42:49.926135    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af\": container with ID starting with 2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af not found: ID does not exist" containerID="2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"
Jul 28 17:42:49.926230 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.926166    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af"} err="failed to get container status \"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af\": rpc error: code = NotFound desc = could not find container \"2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af\": container with ID starting with 2721ff9f6f67c5faba787ca6c691a571d9a0390393587332724cf2dd0f8cb4af not found: ID does not exist"
Jul 28 17:42:49.926230 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.926190    2573 scope.go:117] "RemoveContainer" containerID="190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"
Jul 28 17:42:49.926421 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:42:49.926406    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32\": container with ID starting with 190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32 not found: ID does not exist" containerID="190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"
Jul 28 17:42:49.926471 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.926430    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32"} err="failed to get container status \"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32\": rpc error: code = NotFound desc = could not find container \"190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32\": container with ID starting with 190ee970c5cfea9576c54ac90a66e76bccfd5537e8b05d2d5ee5d43d7e051c32 not found: ID does not exist"
Jul 28 17:42:49.931900 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.931877    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:42:49.937224 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:49.937205    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d1fc0-predictor-6d9fcbc679-5w7nd"]
Jul 28 17:42:51.605355 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:51.605326    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" path="/var/lib/kubelet/pods/8516fcf2-f6ee-4544-9dd7-56ccda7afae2/volumes"
Jul 28 17:42:53.911150 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:53.911122    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:42:53.911644 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:42:53.911622    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:43:03.911805 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:03.911765    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:43:13.912589 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:13.912551    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:43:23.911787 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:23.911747    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:43:33.912170 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:33.912140    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:43:41.989461 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:41.989383    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.068227 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.068181    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.093343 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.093310    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:43:42.093637 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.093607    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" containerID="cri-o://8518573d26344edb6b969ad75806979da1c99e0bfa1364ea9de7c205da73bac9" gracePeriod=30
Jul 28 17:43:42.093743 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.093667    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kube-rbac-proxy" containerID="cri-o://bdc9d41332a0b09f0fdf85b99d19a6258cac8928f880b4bab723aec6c253714c" gracePeriod=30
Jul 28 17:43:42.141143 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.141114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.157798 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.157768    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 17:43:42.158134 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158119    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container"
Jul 28 17:43:42.158222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158137    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container"
Jul 28 17:43:42.158222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158153    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kube-rbac-proxy"
Jul 28 17:43:42.158222 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158162    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kube-rbac-proxy"
Jul 28 17:43:42.158375 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158250    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kserve-container"
Jul 28 17:43:42.158375 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.158261    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8516fcf2-f6ee-4544-9dd7-56ccda7afae2" containerName="kube-rbac-proxy"
Jul 28 17:43:42.161409 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.161389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.164693 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.164672    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dc16-kube-rbac-proxy-sar-config\""
Jul 28 17:43:42.164812 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.164794    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dc16-predictor-serving-cert\""
Jul 28 17:43:42.172889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.172862    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 17:43:42.205504 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.205477    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.210670 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.210649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.210753 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.210690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncsx\" (UniqueName: \"kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.210811 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.210755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.270144 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.270037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.311739 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.311707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.311739 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.311744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncsx\" (UniqueName: \"kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.312042 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.311792    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.312042 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:43:42.311904    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-serving-cert: secret "success-200-isvc-0dc16-predictor-serving-cert" not found
Jul 28 17:43:42.312042 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:43:42.311973    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls podName:f54c01fe-90bb-420a-9455-8577a038e7b4 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:42.811951648 +0000 UTC m=+2065.739873285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls") pod "success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" (UID: "f54c01fe-90bb-420a-9455-8577a038e7b4") : secret "success-200-isvc-0dc16-predictor-serving-cert" not found
Jul 28 17:43:42.312562 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.312544    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.321363 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.321339    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncsx\" (UniqueName: \"kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.415496 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.415468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.594238 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.594162    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.816912 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.816874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:42.819263 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:42.819232    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") pod \"success-200-isvc-0dc16-predictor-66967b5b55-dpcm5\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:43.071331 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:43.071286    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:43.106076 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:43.106021    2573 generic.go:356] "Generic (PLEG): container finished" podID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerID="bdc9d41332a0b09f0fdf85b99d19a6258cac8928f880b4bab723aec6c253714c" exitCode=2
Jul 28 17:43:43.106210 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:43.106081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerDied","Data":"bdc9d41332a0b09f0fdf85b99d19a6258cac8928f880b4bab723aec6c253714c"}
Jul 28 17:43:43.197472 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:43.197439    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 17:43:43.201158 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:43:43.201129    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54c01fe_90bb_420a_9455_8577a038e7b4.slice/crio-0df13fcf8421aa429041d38372aba56f7d3a36d0ef3390ec2e4f0ebb0891b478 WatchSource:0}: Error finding container 0df13fcf8421aa429041d38372aba56f7d3a36d0ef3390ec2e4f0ebb0891b478: Status 404 returned error can't find the container with id 0df13fcf8421aa429041d38372aba56f7d3a36d0ef3390ec2e4f0ebb0891b478
Jul 28 17:43:44.110886 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:44.110855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerStarted","Data":"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"}
Jul 28 17:43:44.111255 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:44.110910    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerStarted","Data":"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"}
Jul 28 17:43:44.111255 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:44.110932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerStarted","Data":"0df13fcf8421aa429041d38372aba56f7d3a36d0ef3390ec2e4f0ebb0891b478"}
Jul 28 17:43:44.111255 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:44.110953    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:44.130541 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:44.130490    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podStartSLOduration=2.130472198 podStartE2EDuration="2.130472198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:44.128929699 +0000 UTC m=+2067.056851366" watchObservedRunningTime="2026-07-28 17:43:44.130472198 +0000 UTC m=+2067.058393853"
Jul 28 17:43:45.115992 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.115957    2573 generic.go:356] "Generic (PLEG): container finished" podID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerID="8518573d26344edb6b969ad75806979da1c99e0bfa1364ea9de7c205da73bac9" exitCode=0
Jul 28 17:43:45.116333 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.116037    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerDied","Data":"8518573d26344edb6b969ad75806979da1c99e0bfa1364ea9de7c205da73bac9"}
Jul 28 17:43:45.116333 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.116311    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:45.117834 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.117806    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:43:45.152346 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.152315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 17:43:45.231841 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.231819    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:43:45.336426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.336339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config\") pod \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") "
Jul 28 17:43:45.336426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.336386    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh6qw\" (UniqueName: \"kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw\") pod \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") "
Jul 28 17:43:45.336426 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.336404    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls\") pod \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\" (UID: \"5dadb749-2e06-4e5c-9cc3-ac42790048f3\") "
Jul 28 17:43:45.336713 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.336686    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dadb749-2e06-4e5c-9cc3-ac42790048f3" (UID: "5dadb749-2e06-4e5c-9cc3-ac42790048f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:45.338443 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.338408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dadb749-2e06-4e5c-9cc3-ac42790048f3" (UID: "5dadb749-2e06-4e5c-9cc3-ac42790048f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:45.338548 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.338459    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw" (OuterVolumeSpecName: "kube-api-access-qh6qw") pod "5dadb749-2e06-4e5c-9cc3-ac42790048f3" (UID: "5dadb749-2e06-4e5c-9cc3-ac42790048f3"). InnerVolumeSpecName "kube-api-access-qh6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:45.437241 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.437209    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:43:45.437241 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.437235    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6qw\" (UniqueName: \"kubernetes.io/projected/5dadb749-2e06-4e5c-9cc3-ac42790048f3-kube-api-access-qh6qw\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:43:45.437241 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:45.437245    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dadb749-2e06-4e5c-9cc3-ac42790048f3-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:43:46.121141 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.121110    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"
Jul 28 17:43:46.121580 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.121140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" event={"ID":"5dadb749-2e06-4e5c-9cc3-ac42790048f3","Type":"ContainerDied","Data":"d31de408af355ee3f49d99983c29d01add1fd42dfe8d72a348dc51b01cdb3539"}
Jul 28 17:43:46.121580 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.121211    2573 scope.go:117] "RemoveContainer" containerID="bdc9d41332a0b09f0fdf85b99d19a6258cac8928f880b4bab723aec6c253714c"
Jul 28 17:43:46.121743 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.121717    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:43:46.128794 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.128776    2573 scope.go:117] "RemoveContainer" containerID="8518573d26344edb6b969ad75806979da1c99e0bfa1364ea9de7c205da73bac9"
Jul 28 17:43:46.147412 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.147391    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:43:46.156782 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.156733    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: i/o timeout"
Jul 28 17:43:46.158672 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.158653    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a513a-predictor-5477b77665-h87n4"]
Jul 28 17:43:46.329315 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.329281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:46.757306 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.757279    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:46.861010 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:46.860982    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:47.603125 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:47.603088    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" path="/var/lib/kubelet/pods/5dadb749-2e06-4e5c-9cc3-ac42790048f3/volumes"
Jul 28 17:43:51.126148 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:51.126116    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:43:51.126576 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:51.126522    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:43:56.402348 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.402273    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:56.508026 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.507992    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:43:56.508353 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.508322    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container" containerID="cri-o://1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17" gracePeriod=30
Jul 28 17:43:56.508489 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.508328    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kube-rbac-proxy" containerID="cri-o://23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309" gracePeriod=30
Jul 28 17:43:56.561799 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.561759    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:43:56.562107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562093    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kube-rbac-proxy"
Jul 28 17:43:56.562107 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562108    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kube-rbac-proxy"
Jul 28 17:43:56.562206 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562126    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container"
Jul 28 17:43:56.562206 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562131    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container"
Jul 28 17:43:56.562206 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562180    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kserve-container"
Jul 28 17:43:56.562206 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.562187    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dadb749-2e06-4e5c-9cc3-ac42790048f3" containerName="kube-rbac-proxy"
Jul 28 17:43:56.565148 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.565123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.568202 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.568181    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-824fb-predictor-serving-cert\""
Jul 28 17:43:56.568556 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.568541    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-824fb-kube-rbac-proxy-sar-config\""
Jul 28 17:43:56.576713 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.576693    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:43:56.724816 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.724786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.725022 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.724849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.725022 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.724915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz76f\" (UniqueName: \"kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.794927 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.794894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:56.826024 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.825991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.826176 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.826068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.826176 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.826116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz76f\" (UniqueName: \"kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.826176 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:43:56.826142    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-824fb-predictor-serving-cert: secret "success-200-isvc-824fb-predictor-serving-cert" not found
Jul 28 17:43:56.826291 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:43:56.826199    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls podName:66d41b88-450f-4ffe-a9a5-8532f9e04040 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:57.326182866 +0000 UTC m=+2080.254104503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls") pod "success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" (UID: "66d41b88-450f-4ffe-a9a5-8532f9e04040") : secret "success-200-isvc-824fb-predictor-serving-cert" not found
Jul 28 17:43:56.826658 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.826641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.836396 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.836376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz76f\" (UniqueName: \"kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:56.899684 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:56.899650    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.114965 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.114892    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.161897 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.161865    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerID="23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309" exitCode=2
Jul 28 17:43:57.161897 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.161892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerDied","Data":"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"}
Jul 28 17:43:57.214036 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.213991    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.330179 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.330147    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:57.332511 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.332486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") pod \"success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") " pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:57.474876 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.474841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:57.603890 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:57.603853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:43:57.606910 ip-10-0-129-82 kubenswrapper[2573]: W0728 17:43:57.606882    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d41b88_450f_4ffe_a9a5_8532f9e04040.slice/crio-34b8a8ae2c34b8da5f6a407feb4473390435ea1ade761aa3827a9a1d149ab098 WatchSource:0}: Error finding container 34b8a8ae2c34b8da5f6a407feb4473390435ea1ade761aa3827a9a1d149ab098: Status 404 returned error can't find the container with id 34b8a8ae2c34b8da5f6a407feb4473390435ea1ade761aa3827a9a1d149ab098
Jul 28 17:43:58.166747 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.166716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerStarted","Data":"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"}
Jul 28 17:43:58.166928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.166771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerStarted","Data":"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"}
Jul 28 17:43:58.166928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.166812    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:58.166928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.166862    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:43:58.166928 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.166892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerStarted","Data":"34b8a8ae2c34b8da5f6a407feb4473390435ea1ade761aa3827a9a1d149ab098"}
Jul 28 17:43:58.168300 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.168274    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:43:58.187578 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.187529    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podStartSLOduration=2.187514939 podStartE2EDuration="2.187514939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:58.185273412 +0000 UTC m=+2081.113195067" watchObservedRunningTime="2026-07-28 17:43:58.187514939 +0000 UTC m=+2081.115436593"
Jul 28 17:43:58.907341 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:58.907302    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 17:43:59.171069 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.171017    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:43:59.754510 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.754485    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:43:59.849996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.849910    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls\") pod \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") "
Jul 28 17:43:59.849996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.849955    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config\") pod \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") "
Jul 28 17:43:59.849996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.849995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlbh4\" (UniqueName: \"kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4\") pod \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\" (UID: \"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe\") "
Jul 28 17:43:59.850391 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.850366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" (UID: "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:59.852151 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.852110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" (UID: "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:59.852301 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.852276    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4" (OuterVolumeSpecName: "kube-api-access-jlbh4") pod "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" (UID: "e8b0b08b-2f90-49b6-9c54-f2cfa08459fe"). InnerVolumeSpecName "kube-api-access-jlbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:59.950777 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.950740    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlbh4\" (UniqueName: \"kubernetes.io/projected/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-api-access-jlbh4\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:43:59.950777 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.950771    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:43:59.950777 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:43:59.950784    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:44:00.176533 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.176499    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerID="1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17" exitCode=0
Jul 28 17:44:00.176685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.176546    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"
Jul 28 17:44:00.176685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.176584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerDied","Data":"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"}
Jul 28 17:44:00.176685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.176639    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z" event={"ID":"e8b0b08b-2f90-49b6-9c54-f2cfa08459fe","Type":"ContainerDied","Data":"603b4f840d7b45d6589b1919524774bbb394381c0d5677b47689894510e8a833"}
Jul 28 17:44:00.176685 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.176670    2573 scope.go:117] "RemoveContainer" containerID="23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"
Jul 28 17:44:00.188209 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.188187    2573 scope.go:117] "RemoveContainer" containerID="1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"
Jul 28 17:44:00.195320 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.195303    2573 scope.go:117] "RemoveContainer" containerID="23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"
Jul 28 17:44:00.195554 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:44:00.195533    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309\": container with ID starting with 23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309 not found: ID does not exist" containerID="23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"
Jul 28 17:44:00.195603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.195561    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309"} err="failed to get container status \"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309\": rpc error: code = NotFound desc = could not find container \"23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309\": container with ID starting with 23c0bfc38da74ee45404ab6bfa3e4088cebc4717be544b0af6049cb70453e309 not found: ID does not exist"
Jul 28 17:44:00.195603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.195576    2573 scope.go:117] "RemoveContainer" containerID="1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"
Jul 28 17:44:00.195763 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:44:00.195748    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17\": container with ID starting with 1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17 not found: ID does not exist" containerID="1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"
Jul 28 17:44:00.195799 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.195766    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17"} err="failed to get container status \"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17\": rpc error: code = NotFound desc = could not find container \"1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17\": container with ID starting with 1a8acbfc7f8cc6ac06f88c079206770f2646e9ea2b4c70f5460836de371d8e17 not found: ID does not exist"
Jul 28 17:44:00.201706 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.201687    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:44:00.207605 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:00.207586    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11994-predictor-5bbcdf967f-9qn7z"]
Jul 28 17:44:01.126571 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:01.126534    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:44:01.600260 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:01.600229    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" path="/var/lib/kubelet/pods/e8b0b08b-2f90-49b6-9c54-f2cfa08459fe/volumes"
Jul 28 17:44:01.955081 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:01.955038    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:04.175371 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:04.175346    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:44:04.175799 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:04.175773    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:44:11.127116 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:11.127072    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:44:14.175845 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:14.175803    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:44:21.126657 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:21.126612    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:44:24.176303 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:24.176258    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:44:31.127897 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:31.127866    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 17:44:34.176667 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:34.176630    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:44:42.233704 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:42.233671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:42.511827 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:42.511743    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:44.176889 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:44.176860    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:44:56.647980 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:56.647943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:56.912964 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:44:56.912921    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:12.132203 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:48:12.132118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:19.020903 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:52:19.020874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:06.115948 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:06.115910    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:11.198383 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.198349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:11.317631 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.317596    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:53:11.317891 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.317866    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" containerID="cri-o://96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468" gracePeriod=30
Jul 28 17:53:11.318012 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.317971    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kube-rbac-proxy" containerID="cri-o://d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb" gracePeriod=30
Jul 28 17:53:11.349252 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.349219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:11.435178 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:11.435145    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:12.156288 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:12.156174    2573 generic.go:356] "Generic (PLEG): container finished" podID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerID="d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb" exitCode=2
Jul 28 17:53:12.156288 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:12.156248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerDied","Data":"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"}
Jul 28 17:53:14.172331 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.172288    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 17:53:14.176665 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.176638    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:53:14.464130 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.464106    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:53:14.564909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.564871    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz76f\" (UniqueName: \"kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f\") pod \"66d41b88-450f-4ffe-a9a5-8532f9e04040\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") "
Jul 28 17:53:14.564909 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.564916    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config\") pod \"66d41b88-450f-4ffe-a9a5-8532f9e04040\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") "
Jul 28 17:53:14.565188 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.564959    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") pod \"66d41b88-450f-4ffe-a9a5-8532f9e04040\" (UID: \"66d41b88-450f-4ffe-a9a5-8532f9e04040\") "
Jul 28 17:53:14.565319 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.565294    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66d41b88-450f-4ffe-a9a5-8532f9e04040" (UID: "66d41b88-450f-4ffe-a9a5-8532f9e04040"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:14.567031 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.567003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f" (OuterVolumeSpecName: "kube-api-access-pz76f") pod "66d41b88-450f-4ffe-a9a5-8532f9e04040" (UID: "66d41b88-450f-4ffe-a9a5-8532f9e04040"). InnerVolumeSpecName "kube-api-access-pz76f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:14.567118 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.567018    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66d41b88-450f-4ffe-a9a5-8532f9e04040" (UID: "66d41b88-450f-4ffe-a9a5-8532f9e04040"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:14.666266 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.666230    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz76f\" (UniqueName: \"kubernetes.io/projected/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-api-access-pz76f\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:53:14.666266 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.666262    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d41b88-450f-4ffe-a9a5-8532f9e04040-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:53:14.666266 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:14.666272    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d41b88-450f-4ffe-a9a5-8532f9e04040-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 17:53:15.168251 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.168208    2573 generic.go:356] "Generic (PLEG): container finished" podID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerID="96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468" exitCode=0
Jul 28 17:53:15.168441 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.168263    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"
Jul 28 17:53:15.168441 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.168280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerDied","Data":"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"}
Jul 28 17:53:15.168441 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.168346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr" event={"ID":"66d41b88-450f-4ffe-a9a5-8532f9e04040","Type":"ContainerDied","Data":"34b8a8ae2c34b8da5f6a407feb4473390435ea1ade761aa3827a9a1d149ab098"}
Jul 28 17:53:15.168441 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.168374    2573 scope.go:117] "RemoveContainer" containerID="d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"
Jul 28 17:53:15.176677 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.176659    2573 scope.go:117] "RemoveContainer" containerID="96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"
Jul 28 17:53:15.183603 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.183586    2573 scope.go:117] "RemoveContainer" containerID="d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"
Jul 28 17:53:15.183892 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:53:15.183873    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb\": container with ID starting with d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb not found: ID does not exist" containerID="d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"
Jul 28 17:53:15.183955 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.183901    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb"} err="failed to get container status \"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb\": rpc error: code = NotFound desc = could not find container \"d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb\": container with ID starting with d4dc05ff033904f749c4c0a3e5cef8f40316fce7feac728a411da60bc2c3ccbb not found: ID does not exist"
Jul 28 17:53:15.183955 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.183919    2573 scope.go:117] "RemoveContainer" containerID="96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"
Jul 28 17:53:15.184300 ip-10-0-129-82 kubenswrapper[2573]: E0728 17:53:15.184281    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468\": container with ID starting with 96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468 not found: ID does not exist" containerID="96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"
Jul 28 17:53:15.184380 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.184304    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468"} err="failed to get container status \"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468\": rpc error: code = NotFound desc = could not find container \"96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468\": container with ID starting with 96293c0c719494c07d04e2325bc410d9b1d2396179c41e02c5d24ce0333d4468 not found: ID does not exist"
Jul 28 17:53:15.192996 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.192974    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:53:15.196611 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.196590    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-824fb-predictor-6f7f8d49f-zmdrr"]
Jul 28 17:53:15.600653 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:53:15.600573    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" path="/var/lib/kubelet/pods/66d41b88-450f-4ffe-a9a5-8532f9e04040/volumes"
Jul 28 17:57:53.581670 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:57:53.581634    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:51.377292 ip-10-0-129-82 kubenswrapper[2573]: I0728 17:58:51.377212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:01.568498 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.568459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:01.725367 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.725331    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 18:01:01.725573 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.725552    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container" containerID="cri-o://68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02" gracePeriod=30
Jul 28 18:01:01.725655 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.725603    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kube-rbac-proxy" containerID="cri-o://2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed" gracePeriod=30
Jul 28 18:01:01.757800 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.757758    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:01.844934 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:01.844893    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:02.819202 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:02.819170    2573 generic.go:356] "Generic (PLEG): container finished" podID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerID="2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed" exitCode=2
Jul 28 18:01:02.819591 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:02.819239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerDied","Data":"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"}
Jul 28 18:01:02.990554 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:02.990512    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:03.091256 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:03.091166    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:04.769248 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.769218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 18:01:04.827865 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.827781    2573 generic.go:356] "Generic (PLEG): container finished" podID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerID="68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02" exitCode=0
Jul 28 18:01:04.827865 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.827833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"
Jul 28 18:01:04.828067 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.827864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerDied","Data":"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"}
Jul 28 18:01:04.828067 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.827921    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5" event={"ID":"f54c01fe-90bb-420a-9455-8577a038e7b4","Type":"ContainerDied","Data":"0df13fcf8421aa429041d38372aba56f7d3a36d0ef3390ec2e4f0ebb0891b478"}
Jul 28 18:01:04.828067 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.827950    2573 scope.go:117] "RemoveContainer" containerID="2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"
Jul 28 18:01:04.835657 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.835617    2573 scope.go:117] "RemoveContainer" containerID="68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"
Jul 28 18:01:04.842708 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.842691    2573 scope.go:117] "RemoveContainer" containerID="2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"
Jul 28 18:01:04.842996 ip-10-0-129-82 kubenswrapper[2573]: E0728 18:01:04.842974    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed\": container with ID starting with 2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed not found: ID does not exist" containerID="2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"
Jul 28 18:01:04.843067 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.843007    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed"} err="failed to get container status \"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed\": rpc error: code = NotFound desc = could not find container \"2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed\": container with ID starting with 2b741726f5dc451a837acb5c1b03d4c4bc06464691c35a90479746f0ec120fed not found: ID does not exist"
Jul 28 18:01:04.843067 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.843025    2573 scope.go:117] "RemoveContainer" containerID="68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"
Jul 28 18:01:04.843290 ip-10-0-129-82 kubenswrapper[2573]: E0728 18:01:04.843272    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02\": container with ID starting with 68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02 not found: ID does not exist" containerID="68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"
Jul 28 18:01:04.843359 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.843306    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02"} err="failed to get container status \"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02\": rpc error: code = NotFound desc = could not find container \"68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02\": container with ID starting with 68c813a11d34ec30e348443bde7cd9419990ae26889a2d3f898088dcbe981b02 not found: ID does not exist"
Jul 28 18:01:04.911147 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.911119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") pod \"f54c01fe-90bb-420a-9455-8577a038e7b4\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") "
Jul 28 18:01:04.911339 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.911163    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config\") pod \"f54c01fe-90bb-420a-9455-8577a038e7b4\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") "
Jul 28 18:01:04.911339 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.911188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tncsx\" (UniqueName: \"kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx\") pod \"f54c01fe-90bb-420a-9455-8577a038e7b4\" (UID: \"f54c01fe-90bb-420a-9455-8577a038e7b4\") "
Jul 28 18:01:04.911530 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.911506    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f54c01fe-90bb-420a-9455-8577a038e7b4" (UID: "f54c01fe-90bb-420a-9455-8577a038e7b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:04.917084 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.913648    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx" (OuterVolumeSpecName: "kube-api-access-tncsx") pod "f54c01fe-90bb-420a-9455-8577a038e7b4" (UID: "f54c01fe-90bb-420a-9455-8577a038e7b4"). InnerVolumeSpecName "kube-api-access-tncsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:04.918224 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:04.918194    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f54c01fe-90bb-420a-9455-8577a038e7b4" (UID: "f54c01fe-90bb-420a-9455-8577a038e7b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:05.012580 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.012534    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54c01fe-90bb-420a-9455-8577a038e7b4-proxy-tls\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 18:01:05.012580 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.012572    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 18:01:05.012580 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.012586    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncsx\" (UniqueName: \"kubernetes.io/projected/f54c01fe-90bb-420a-9455-8577a038e7b4-kube-api-access-tncsx\") on node \"ip-10-0-129-82.ec2.internal\" DevicePath \"\""
Jul 28 18:01:05.150906 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.150866    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 18:01:05.155547 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.155520    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc16-predictor-66967b5b55-dpcm5"]
Jul 28 18:01:05.600972 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:05.600930    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" path="/var/lib/kubelet/pods/f54c01fe-90bb-420a-9455-8577a038e7b4/volumes"
Jul 28 18:01:31.284921 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:31.284883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:31.445904 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:31.445873    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:31.531536 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:31.531497    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:35.253915 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:35.253877    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b6j9f_de7c57c7-30f7-4d87-ae8a-1c548da27068/global-pull-secret-syncer/0.log"
Jul 28 18:01:35.342520 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:35.342489    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log"
Jul 28 18:01:35.435482 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:35.435448    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-82.ec2.internal_7134973092cf17d72d638832558ef84a/haproxy/0.log"
Jul 28 18:01:37.045345 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:37.045302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:37.147174 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:37.147141    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6rlw_3708ca42-b526-496a-9c39-bf295698f0f6/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:40.654521 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:40.654437    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/node-exporter/0.log"
Jul 28 18:01:40.678115 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:40.678080    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/kube-rbac-proxy/0.log"
Jul 28 18:01:40.708823 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:40.708800    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4gj76_150c1b4e-b984-4b42-9b0c-9d04f3641320/init-textfile/0.log"
Jul 28 18:01:41.319032 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:41.319002    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/telemeter-client/0.log"
Jul 28 18:01:41.344249 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:41.344219    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/reload/0.log"
Jul 28 18:01:41.366882 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:41.366853    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f8cc7554-g746n_08029731-101b-4166-a46c-71bae3782f8c/kube-rbac-proxy/0.log"
Jul 28 18:01:43.838065 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.838021    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-677f74f889-rzf5t_7e297f07-650a-444d-818f-d7d82eac2a9a/console/0.log"
Jul 28 18:01:43.991966 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.991934    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"]
Jul 28 18:01:43.992246 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992233    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992248    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992257    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992262    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992270    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992276    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992291 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992289    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992298    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992308    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992315    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992323    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992328    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992378    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992386    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992393    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992400    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c01fe-90bb-420a-9455-8577a038e7b4" containerName="kserve-container"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992405    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b0b08b-2f90-49b6-9c54-f2cfa08459fe" containerName="kube-rbac-proxy"
Jul 28 18:01:43.992501 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.992411    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d41b88-450f-4ffe-a9a5-8532f9e04040" containerName="kserve-container"
Jul 28 18:01:43.995274 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.995257    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:43.998025 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.998001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nsx4t\"/\"default-dockercfg-qqmtf\""
Jul 28 18:01:43.998189 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.998042    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsx4t\"/\"kube-root-ca.crt\""
Jul 28 18:01:43.999492 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:43.999471    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsx4t\"/\"openshift-service-ca.crt\""
Jul 28 18:01:44.004133 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.004110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"]
Jul 28 18:01:44.123315 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.123219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-proc\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.123315 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.123271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-podres\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.123315 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.123311    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-sys\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.123537 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.123336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s84c\" (UniqueName: \"kubernetes.io/projected/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-kube-api-access-4s84c\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.123537 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.123375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-lib-modules\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223770 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-proc\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223898 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-podres\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223898 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-sys\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223898 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s84c\" (UniqueName: \"kubernetes.io/projected/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-kube-api-access-4s84c\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223898 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-proc\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.223898 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-lib-modules\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.224157 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-sys\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.224157 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.223988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-podres\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.224157 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.224127    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-lib-modules\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.233029 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.233004    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s84c\" (UniqueName: \"kubernetes.io/projected/b0e32237-9f2d-44f0-b150-e1ba5a9979d5-kube-api-access-4s84c\") pod \"perf-node-gather-daemonset-bspg6\" (UID: \"b0e32237-9f2d-44f0-b150-e1ba5a9979d5\") " pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.305371 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.305332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.443552 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.443528    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"]
Jul 28 18:01:44.446396 ip-10-0-129-82 kubenswrapper[2573]: W0728 18:01:44.446366    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb0e32237_9f2d_44f0_b150_e1ba5a9979d5.slice/crio-a3295ec009b9923fcde04fb5ece35b249146b2441f37a19a0ea2f461ccde326e WatchSource:0}: Error finding container a3295ec009b9923fcde04fb5ece35b249146b2441f37a19a0ea2f461ccde326e: Status 404 returned error can't find the container with id a3295ec009b9923fcde04fb5ece35b249146b2441f37a19a0ea2f461ccde326e
Jul 28 18:01:44.972108 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.972074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6" event={"ID":"b0e32237-9f2d-44f0-b150-e1ba5a9979d5","Type":"ContainerStarted","Data":"98219b252d37b2bf1e7bf094be9ffef29f98417d137b5358f6344a16253bb7fa"}
Jul 28 18:01:44.972628 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.972135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6" event={"ID":"b0e32237-9f2d-44f0-b150-e1ba5a9979d5","Type":"ContainerStarted","Data":"a3295ec009b9923fcde04fb5ece35b249146b2441f37a19a0ea2f461ccde326e"}
Jul 28 18:01:44.972628 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.972182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:44.992208 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:44.992161    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6" podStartSLOduration=1.992144702 podStartE2EDuration="1.992144702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:44.989895688 +0000 UTC m=+3147.917817356" watchObservedRunningTime="2026-07-28 18:01:44.992144702 +0000 UTC m=+3147.920066361"
Jul 28 18:01:45.391862 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:45.391771    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bswg7_706624fd-df7d-4366-994a-c1a4d310a483/dns/0.log"
Jul 28 18:01:45.427364 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:45.427332    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bswg7_706624fd-df7d-4366-994a-c1a4d310a483/kube-rbac-proxy/0.log"
Jul 28 18:01:45.535345 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:45.535310    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cs2zt_8860ffcf-5fdc-47f3-af20-66a41e9bd479/dns-node-resolver/0.log"
Jul 28 18:01:46.077627 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:46.077592    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d8fc4f564-xgb8k_2be1ca24-53fb-40c9-bc7f-2b83ff9014c5/registry/0.log"
Jul 28 18:01:46.118184 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:46.118154    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs5m9_9605dad1-12b3-4da6-ac7e-a08f7654b6ed/node-ca/0.log"
Jul 28 18:01:47.325248 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:47.325216    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lkktt_8b9a6911-897c-43e0-81f0-56627ef9c2f6/serve-healthcheck-canary/0.log"
Jul 28 18:01:47.873716 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:47.873687    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7trjn_a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff/kube-rbac-proxy/0.log"
Jul 28 18:01:47.898584 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:47.898556    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7trjn_a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff/exporter/0.log"
Jul 28 18:01:47.924817 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:47.924790    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7trjn_a7ebd6cb-c7a2-4b1f-b3d9-291980fe1bff/extractor/0.log"
Jul 28 18:01:50.235531 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:50.235501    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cd4d44c67-f9jbv_bde3300b-b272-4120-a941-1ad3b1598a39/manager/0.log"
Jul 28 18:01:50.467292 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:50.467255    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-tcs7p_73f5709b-4a02-445d-bf1b-1e88f2dab521/manager/0.log"
Jul 28 18:01:50.548888 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:50.548800    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4mpbj_8e12c82f-e6d5-4447-90ad-6684de53cd4e/seaweedfs/0.log"
Jul 28 18:01:50.985237 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:50.985207    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nsx4t/perf-node-gather-daemonset-bspg6"
Jul 28 18:01:56.321663 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.321631    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:01:56.345445 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.345414    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/egress-router-binary-copy/0.log"
Jul 28 18:01:56.368666 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.368640    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/cni-plugins/0.log"
Jul 28 18:01:56.391682 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.391649    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/bond-cni-plugin/0.log"
Jul 28 18:01:56.416785 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.416760    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/routeoverride-cni/0.log"
Jul 28 18:01:56.441680 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.441651    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/whereabouts-cni-bincopy/0.log"
Jul 28 18:01:56.465892 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.465862    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6brt_8067b205-077a-4371-a852-08575e036681/whereabouts-cni/0.log"
Jul 28 18:01:56.705698 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.705668    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m82sq_4bd7d11a-7276-49b2-8853-48eef0b3b70a/kube-multus/0.log"
Jul 28 18:01:56.810868 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.810839    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lc55j_cae1727b-dacc-439b-b01d-81f6b2bdc431/network-metrics-daemon/0.log"
Jul 28 18:01:56.833565 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:56.833535    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lc55j_cae1727b-dacc-439b-b01d-81f6b2bdc431/kube-rbac-proxy/0.log"
Jul 28 18:01:57.665150 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.665125    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/ovn-controller/0.log"
Jul 28 18:01:57.685089 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.685059    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/ovn-acl-logging/0.log"
Jul 28 18:01:57.720576 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.720533    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/kube-rbac-proxy-node/0.log"
Jul 28 18:01:57.752507 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.752470    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:01:57.774203 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.774171    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/northd/0.log"
Jul 28 18:01:57.807637 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.807608    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/nbdb/0.log"
Jul 28 18:01:57.833858 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.833828    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/sbdb/0.log"
Jul 28 18:01:57.901162 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:57.901126    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nlq27_c681d353-73da-48e0-9cbc-adfe172ed5f7/ovnkube-controller/0.log"
Jul 28 18:01:59.425217 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:01:59.425186    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-t4wgv_17889c5b-ad0a-48cc-aa55-ddc8d356f448/network-check-target-container/0.log"
Jul 28 18:02:00.425313 ip-10-0-129-82 kubenswrapper[2573]: I0728 18:02:00.425282    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dht2j_c55a2e30-9526-43fb-922b-998c5eff59fa/iptables-alerter/0.log"