Jul 31 23:44:32.442089 ip-10-0-132-129 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 23:44:32.442098 ip-10-0-132-129 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 23:44:32.442104 ip-10-0-132-129 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 23:44:32.442321 ip-10-0-132-129 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 23:44:42.618443 ip-10-0-132-129 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 23:44:42.618458 ip-10-0-132-129 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9d0e5b705ac44c918e9beea3b14777f9 --
Jul 31 23:46:13.430510 ip-10-0-132-129 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:46:13.884371 ip-10-0-132-129 kubenswrapper[2536]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:13.884371 ip-10-0-132-129 kubenswrapper[2536]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:46:13.884371 ip-10-0-132-129 kubenswrapper[2536]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:13.884371 ip-10-0-132-129 kubenswrapper[2536]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:46:13.884371 ip-10-0-132-129 kubenswrapper[2536]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:13.885659 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.885566    2536 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893903    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893927    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893931    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893935    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893938    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893941    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.893932 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893943    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893946    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893949    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893952    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893955    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893957    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893960    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893962    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893965    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893968    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893970    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893973    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893975    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893979    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893982    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893985    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893988    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893991    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893993    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.894195 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893996    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.893998    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894001    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894003    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894006    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894009    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894012    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894019    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894022    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894025    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894028    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894031    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894033    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894036    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894039    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894042    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894045    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894048    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894051    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894053    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.894676 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894055    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894058    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894061    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894064    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894066    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894070    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894073    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894075    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894078    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894082    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894085    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894088    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894091    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894094    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894096    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894104    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894107    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894110    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894112    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894115    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.895158 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894118    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894120    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894123    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894126    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894129    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894131    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894137    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894142    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894145    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894149    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894151    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894154    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894157    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894159    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894163    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894165    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894168    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894170    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894174    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894177    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.895683 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894180    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894182    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894185    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894187    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894190    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894193    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894197    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894202    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894207    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894221    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894224    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894228    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894230    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894233    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894975    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894982    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894985    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894988    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894990    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.896163 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894994    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.894997    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895000    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895003    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895006    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895009    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895012    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895014    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895017    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895020    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895023    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895025    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895028    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895031    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895034    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895037    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895039    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895042    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895044    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895047    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.896637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895049    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895052    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895054    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895057    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895060    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895063    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895065    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895068    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895070    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895073    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895076    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895078    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895081    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895084    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895087    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895089    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895092    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895095    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895097    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895099    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.897139 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895102    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895105    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895107    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895109    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895112    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895117    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895120    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895124    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895126    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895129    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895132    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895136    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895139    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895142    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895145    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895147    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895151    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895154    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895157    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.897637 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895160    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895163    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895166    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895168    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895171    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895174    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895176    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895179    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895182    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895185    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895188    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895190    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895193    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895196    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895198    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895201    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895203    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895206    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895208    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895230    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.898151 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895233    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895236    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895238    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895241    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895244    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895246    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895249    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895253    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895257    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895261    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895264    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895267    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895269    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895272    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895274    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895697    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895703    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895706    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895709    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.898658 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895712    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895716    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895720    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895723    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895726    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895729    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895731    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895734    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895737    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895739    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895742    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895744    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895748    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895750    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895753    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895756    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895758    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895761    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895763    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.899124 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895766    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895769    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895771    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895774    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895777    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895780    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895782    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895785    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895788    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895790    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895793    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895795    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895798    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895800    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895803    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895805    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895808    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895810    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895813    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895815    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.899608 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895818    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895820    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895823    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895826    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895829    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895831    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895834    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895836    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895839    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895841    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895844    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895846    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895849    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895851    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895854    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895859    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895862    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895865    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895867    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895870    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.900090 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895873    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895875    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895878    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895880    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895883    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895886    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895888    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895891    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895893    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895896    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895901    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895904    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895907    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895909    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895913    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895917    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895919    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895922    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895925    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.900592 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895931    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895935    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895937    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895940    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895943    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895945    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895948    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895950    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895957    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895960    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895962    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895965    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895967    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895970    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895972    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895975    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.895978    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896056    2536 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896064    2536 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896071    2536 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896075    2536 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:46:13.901043 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896079    2536 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896083    2536 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896087    2536 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896092    2536 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896095    2536 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896098    2536 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896101    2536 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896105    2536 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896108    2536 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896111    2536 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896114    2536 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896117    2536 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896123    2536 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896126    2536 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896131    2536 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896134    2536 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896137    2536 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896140    2536 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896143    2536 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896149    2536 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896152    2536 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896155    2536 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896159    2536 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896162    2536 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:46:13.901559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896165    2536 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896168    2536 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896172    2536 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896175    2536 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896179    2536 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896182    2536 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896185    2536 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896188    2536 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896191    2536 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896194    2536 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896200    2536 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896203    2536 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896206    2536 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896222    2536 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896225    2536 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896234    2536 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896237    2536 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896240    2536 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896243    2536 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896247    2536 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896250    2536 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896254    2536 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896257    2536 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896260    2536 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896263    2536 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:46:13.902135 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896266    2536 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896271    2536 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896276    2536 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896279    2536 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896282    2536 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896285    2536 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896288    2536 flags.go:64] FLAG: --help="false"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896291    2536 flags.go:64] FLAG: --hostname-override="ip-10-0-132-129.ec2.internal"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896294    2536 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896297    2536 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896301    2536 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896304    2536 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896308    2536 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896311    2536 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896313    2536 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896316    2536 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896319    2536 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896322    2536 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896325    2536 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896328    2536 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896331    2536 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896334    2536 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896337    2536 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896340    2536 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:46:13.902762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896343    2536 flags.go:64] FLAG: --lock-file=""
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896345    2536 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896349    2536 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896351    2536 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896357    2536 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896360    2536 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896363    2536 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896366    2536 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896369    2536 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896372    2536 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896376    2536 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896379    2536 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896383    2536 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896386    2536 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896391    2536 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896394    2536 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896397    2536 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896400    2536 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896404    2536 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896407    2536 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896410    2536 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896413    2536 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896421    2536 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896424    2536 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896427    2536 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:46:13.903357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896430    2536 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896433    2536 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896439    2536 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896442    2536 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896445    2536 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896448    2536 flags.go:64] FLAG: --port="10250"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896451    2536 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896454    2536 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dbe42af97f37b2cc"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896457    2536 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896460    2536 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896463    2536 flags.go:64] FLAG: --register-node="true"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896466    2536 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896470    2536 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896473    2536 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896477    2536 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896479    2536 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896483    2536 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896486    2536 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896490    2536 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896493    2536 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896496    2536 flags.go:64] FLAG: --runonce="false"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896499    2536 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896502    2536 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896505    2536 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896508    2536 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:46:13.903969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896511    2536 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896514    2536 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896517    2536 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896520    2536 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896523    2536 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896526    2536 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896529    2536 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896532    2536 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896535    2536 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896538    2536 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896540    2536 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896546    2536 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896548    2536 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896551    2536 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896556    2536 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896559    2536 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896561    2536 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896564    2536 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896567    2536 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896570    2536 flags.go:64] FLAG: --v="2"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896574    2536 flags.go:64] FLAG: --version="false"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896579    2536 flags.go:64] FLAG: --vmodule=""
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896583    2536 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.896586    2536 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896712    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.904586 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896718    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896721    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896724    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896727    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896730    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896732    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896735    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896738    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896742    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896746    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896749    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896752    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896755    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896757    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896760    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896763    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896765    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896768    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896771    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.905204 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896773    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896776    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896779    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896782    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896785    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896787    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896790    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896792    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896795    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896797    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896806    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896809    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896811    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896815    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896818    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896820    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896823    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896826    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896836    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896839    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.905679 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896843    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896846    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896849    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896851    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896854    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896856    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896859    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896861    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896865    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896867    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896870    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896873    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896875    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896878    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896881    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896883    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896886    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896888    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896891    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896893    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.906162 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896896    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896898    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896901    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896907    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896910    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896914    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896917    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896919    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896922    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896924    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896927    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896932    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896937    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896940    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896942    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896945    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896948    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896950    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896953    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.906774 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896955    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896958    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896961    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896964    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896983    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896987    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896990    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896993    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896996    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.896998    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897002    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897004    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897007    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897009    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897012    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897014    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897017    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897021    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897026    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.907282 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.897028    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.898123    2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905146    2536 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905166    2536 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905181    2536 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905187    2536 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905265    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905270    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905274    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905277    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905280    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905283    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905286    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905289    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905292    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.907749 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905295    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905298    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905301    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905304    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905307    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905310    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905313    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905316    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905320    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905324    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905327    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905330    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905333    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905335    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905338    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905342    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905344    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905347    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905350    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.908128 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905353    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905356    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905359    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905362    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905364    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905367    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905370    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905373    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905375    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905378    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905381    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905383    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905386    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905388    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905392    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905394    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905397    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905399    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905402    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905405    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.908606 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905407    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905411    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905415    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905418    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905421    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905424    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905427    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905430    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905433    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905436    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905438    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905441    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905444    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905448    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905451    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905453    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905456    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905458    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905462    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.909098 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905466    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905468    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905471    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905474    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905477    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905479    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905482    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905485    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905487    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905490    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905492    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905496    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905498    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905501    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905503    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905506    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905508    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905511    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905514    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905516    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.909571 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905519    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905522    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905524    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905527    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905530    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905533    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905536    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905539    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905542    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905545    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905547    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905550    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905556    2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905679    2536 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905684    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:13.910053 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905687    2536 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905690    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905693    2536 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905695    2536 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905698    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905701    2536 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905705    2536 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905708    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905710    2536 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905713    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905715    2536 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905718    2536 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905722    2536 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905726    2536 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905729    2536 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905732    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905734    2536 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905737    2536 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905740    2536 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:13.910510 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905742    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905745    2536 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905748    2536 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905751    2536 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905754    2536 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905758    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905760    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905763    2536 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905766    2536 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905768    2536 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905771    2536 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905773    2536 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905777    2536 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905779    2536 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905782    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905785    2536 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905787    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905790    2536 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905792    2536 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:13.911001 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905795    2536 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905798    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905800    2536 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905803    2536 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905806    2536 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905808    2536 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905811    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905813    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905816    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905819    2536 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905821    2536 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905824    2536 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905826    2536 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905830    2536 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905832    2536 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905835    2536 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905838    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905841    2536 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905844    2536 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905846    2536 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:13.911504 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905849    2536 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905852    2536 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905854    2536 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905857    2536 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905885    2536 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905888    2536 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905891    2536 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905894    2536 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905897    2536 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905900    2536 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905903    2536 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905906    2536 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905909    2536 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905912    2536 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905916    2536 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905920    2536 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905923    2536 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905925    2536 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905928    2536 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905930    2536 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:13.911983 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905933    2536 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905936    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905939    2536 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905942    2536 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905945    2536 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905948    2536 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905950    2536 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905953    2536 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905955    2536 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905962    2536 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905966    2536 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905968    2536 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905971    2536 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905973    2536 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905977    2536 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905981    2536 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905984    2536 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905987    2536 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:13.912473 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:13.905990    2536 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.905995    2536 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.906120    2536 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.908315    2536 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.909917    2536 server.go:1019] "Starting client certificate rotation"
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.910018    2536 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.910091    2536 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:13.912909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.912654    2536 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:13.917881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.917862    2536 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:13.930548 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.930527    2536 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:46:13.932717 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.932693    2536 log.go:25] "Validated CRI v1 image API"
Jul 31 23:46:13.933578 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.933561    2536 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:46:13.937092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.937071    2536 fs.go:135] Filesystem UUIDs: map[49509b8a-aeae-47cf-aae9-84a848b13111:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d311e4dd-23c2-41b4-9c36-1d8455862d3b:/dev/nvme0n1p3]
Jul 31 23:46:13.937189 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.937090    2536 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:46:13.942489 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.942470    2536 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:13.942564 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.942399    2536 manager.go:216] Machine: {Timestamp:2026-07-31 23:46:13.941037728 +0000 UTC m=+0.379505996 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102466 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f8e5097b1005c72ae81c25df3ee8 SystemUUID:ec28f8e5-097b-1005-c72a-e81c25df3ee8 BootID:9d0e5b70-5ac4-4c91-8e9b-eea3b14777f9 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e6:86:1d:e1:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e6:86:1d:e1:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:da:68:1f:ed:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:46:13.942626 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.942568    2536 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:46:13.942674 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.942661    2536 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:46:13.945091 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.945058    2536 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:46:13.945262 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.945094    2536 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-129.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:46:13.946123 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.946111    2536 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:46:13.946159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.946127    2536 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:46:13.946159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.946138    2536 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:13.946159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.946154    2536 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:13.947156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.947145    2536 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:46:13.951566 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951547    2536 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:13.951704 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951695    2536 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:46:13.951735 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951727    2536 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:46:13.951762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951746    2536 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:46:13.951794 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951762    2536 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:46:13.951794 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951775    2536 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:46:13.951794 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.951785    2536 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:46:13.952176 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.952165    2536 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:13.952208 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.952185    2536 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:13.954733 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.954711    2536 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:46:13.956499 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.956485    2536 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:46:13.956536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.956516    2536 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:46:13.957694 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957678    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:46:13.957740 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957707    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:46:13.957740 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957720    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:46:13.957740 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957732    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957748    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957760    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957775    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957787    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957803    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:46:13.957823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957815    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:46:13.957975 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957844    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:46:13.957975 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.957856    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:46:13.959352 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.959338    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:46:13.959405 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.959354    2536 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:46:13.962660 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.962645    2536 server.go:1268] "Started kubelet"
Jul 31 23:46:13.963106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.962958    2536 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:46:13.963524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.963459    2536 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:46:13.963600 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.963554    2536 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:46:13.963637 ip-10-0-132-129 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:46:13.968810 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.968785    2536 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:46:13.973269 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.968929    2536 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-129.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:46:13.973269 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.969325    2536 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:46:13.973269 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.969451    2536 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:46:13.973269 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.970311    2536 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:46:13.973269 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.971721    2536 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:46:13.974018 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.974004    2536 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:46:13.974070 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.974016    2536 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:13.974126 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.974091    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:13.974631 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.974600    2536 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:46:13.974790 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.974777    2536 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:46:13.975021 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975001    2536 factory.go:55] Registering systemd factory
Jul 31 23:46:13.975091 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975026    2536 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:46:13.975091 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975009    2536 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:46:13.975183 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975160    2536 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:46:13.975183 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975170    2536 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:46:13.975386 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975365    2536 factory.go:153] Registering CRI-O factory
Jul 31 23:46:13.975386 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975384    2536 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:46:13.975547 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975430    2536 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:46:13.975547 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975452    2536 factory.go:103] Registering Raw factory
Jul 31 23:46:13.975547 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975466    2536 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:46:13.975802 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.975785    2536 manager.go:349] Starting recovery of all containers
Jul 31 23:46:13.976789 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.975501    2536 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-129.ec2.internal.18c78499efbbbc68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-129.ec2.internal,UID:ip-10-0-132-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-129.ec2.internal,},FirstTimestamp:2026-07-31 23:46:13.962611816 +0000 UTC m=+0.401080087,LastTimestamp:2026-07-31 23:46:13.962611816 +0000 UTC m=+0.401080087,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-129.ec2.internal,}"
Jul 31 23:46:13.981010 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.980982    2536 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:46:13.981122 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.981032    2536 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:46:13.983969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.983950    2536 manager.go:354] Recovery completed
Jul 31 23:46:13.988314 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.988296    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:13.990822 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.990801    2536 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4brm"
Jul 31 23:46:13.991736 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.991715    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:13.991832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.991749    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:13.991832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.991762    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:13.992788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.992771    2536 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:46:13.992788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.992785    2536 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:46:13.992931 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.992804    2536 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:13.992931 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.992815    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:13.993789 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:13.993715    2536 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-129.ec2.internal.18c78499f17824b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-129.ec2.internal,UID:ip-10-0-132-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-129.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-129.ec2.internal,},FirstTimestamp:2026-07-31 23:46:13.991736504 +0000 UTC m=+0.430204777,LastTimestamp:2026-07-31 23:46:13.991736504 +0000 UTC m=+0.430204777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-129.ec2.internal,}"
Jul 31 23:46:13.993899 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.993802    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:13.993899 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.993829    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:13.993899 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.993843    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:13.994225 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.994194    2536 policy_none.go:49] "None policy: Start"
Jul 31 23:46:13.994290 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.994229    2536 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:46:13.994290 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.994243    2536 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:46:13.995234 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.995205    2536 policy_none.go:47] "Start"
Jul 31 23:46:13.996788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:13.996770    2536 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4brm"
Jul 31 23:46:14.024890 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.024870    2536 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:46:14.025006 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.024913    2536 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:46:14.025006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.024924    2536 server.go:87] "Starting device plugin registration server"
Jul 31 23:46:14.025193 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.025180    2536 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:46:14.025257 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.025196    2536 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:46:14.025310 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.025293    2536 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:46:14.025681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.025369    2536 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:46:14.025681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.025381    2536 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:46:14.026017 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.025993    2536 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:46:14.026071 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.026057    2536 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.080277 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.080232    2536 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:46:14.082512 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.081509    2536 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:46:14.082512 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.081548    2536 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:46:14.082512 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.081571    2536 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:46:14.082512 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.081607    2536 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:46:14.084251 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.084229    2536 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:14.125459 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.125426    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:14.126816 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.126799    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:14.126886 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.126835    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:14.126886 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.126845    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:14.126886 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.126870    2536 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.139090 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.139033    2536 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.139090 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.139060    2536 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-129.ec2.internal\": node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.176875 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.176699    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.181735 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.181706    2536 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"]
Jul 31 23:46:14.181820 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.181783    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:14.182789 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.182771    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:14.182871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.182807    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:14.182871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.182823    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:14.185193 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.185177    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:14.185311 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.185294    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.185368 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.185329    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:14.185994 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.185976    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:14.185994 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.185985    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:14.186111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.186008    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:14.186111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.186009    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:14.186111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.186022    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:14.186111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.186028    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:14.188176 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.188162    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.188257 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.188192    2536 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:14.188984 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.188963    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:14.189068 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.188992    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:14.189068 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.189007    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:14.216043 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.216019    2536 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-129.ec2.internal\" not found" node="ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.220629 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.220612    2536 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-129.ec2.internal\" not found" node="ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.276960 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.276929    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.276960 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.276960    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e48bbd57a2cb073c5f8cf46fee7320a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-129.ec2.internal\" (UID: \"8e48bbd57a2cb073c5f8cf46fee7320a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.277175 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.276987    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.277175 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.277005    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377771 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.377742    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.377871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377773    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e48bbd57a2cb073c5f8cf46fee7320a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-129.ec2.internal\" (UID: \"8e48bbd57a2cb073c5f8cf46fee7320a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377799    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377815    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377976 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377880    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e48bbd57a2cb073c5f8cf46fee7320a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-129.ec2.internal\" (UID: \"8e48bbd57a2cb073c5f8cf46fee7320a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377976 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377924    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.377976 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.377882    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4dc6b340f9875c6e96d1cfc5fb33dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal\" (UID: \"9f4dc6b340f9875c6e96d1cfc5fb33dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.478513 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.478437    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:14.518657 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.518632    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.524996 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:14.524978    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:14.578918 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:14.578877    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.152570 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.152534    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.253157 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.253052    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.353514 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.353468    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.382685 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.382652    2536 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:46:16.383139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.382809    2536 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:16.454239 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.454207    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.474990 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.474942    2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:41:15 +0000 UTC" deadline="2028-02-04 09:23:31.697124362 +0000 UTC"
Jul 31 23:46:16.474990 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.474979    2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13257h37m15.222149626s"
Jul 31 23:46:16.540806 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:16.540766    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e48bbd57a2cb073c5f8cf46fee7320a.slice/crio-848670cb3fde26a161731f90ec8363db590fa4bfe535c7a8d8e21af3fbc82359 WatchSource:0}: Error finding container 848670cb3fde26a161731f90ec8363db590fa4bfe535c7a8d8e21af3fbc82359: Status 404 returned error can't find the container with id 848670cb3fde26a161731f90ec8363db590fa4bfe535c7a8d8e21af3fbc82359
Jul 31 23:46:16.541116 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:16.541090    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4dc6b340f9875c6e96d1cfc5fb33dd.slice/crio-d4207ac720407a7b40bbf0ac4dbcaf92e6de7a6bdca50d457f43c5b0382669ee WatchSource:0}: Error finding container d4207ac720407a7b40bbf0ac4dbcaf92e6de7a6bdca50d457f43c5b0382669ee: Status 404 returned error can't find the container with id d4207ac720407a7b40bbf0ac4dbcaf92e6de7a6bdca50d457f43c5b0382669ee
Jul 31 23:46:16.547165 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.547145    2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:46:16.554848 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.554817    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.557893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.557848    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal" event={"ID":"8e48bbd57a2cb073c5f8cf46fee7320a","Type":"ContainerStarted","Data":"848670cb3fde26a161731f90ec8363db590fa4bfe535c7a8d8e21af3fbc82359"}
Jul 31 23:46:16.558962 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.558942    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal" event={"ID":"9f4dc6b340f9875c6e96d1cfc5fb33dd","Type":"ContainerStarted","Data":"d4207ac720407a7b40bbf0ac4dbcaf92e6de7a6bdca50d457f43c5b0382669ee"}
Jul 31 23:46:16.655173 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.655132    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.755555 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.755520    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.855942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.855861    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.943973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.943860    2536 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:16.955980 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:16.955937    2536 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-129.ec2.internal\" not found"
Jul 31 23:46:16.977524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:16.977479    2536 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:17.021580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.021548    2536 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:17.047693 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.047664    2536 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:17.060399 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.060275    2536 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:17.061479 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.061459    2536 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"
Jul 31 23:46:17.069653 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.069545    2536 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:17.426054 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.426018    2536 apiserver.go:52] "Watching apiserver"
Jul 31 23:46:17.435753 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.435719    2536 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:46:17.436090 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.436063    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6fnf7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-d9cnv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz","openshift-dns/node-resolver-zlm46","openshift-image-registry/node-ca-bmzcw","openshift-multus/multus-additional-cni-plugins-n49r5","openshift-multus/multus-lv99k","openshift-multus/network-metrics-daemon-zq5g7","openshift-network-diagnostics/network-check-target-8kmsg","openshift-network-operator/iptables-alerter-x9kdv","kube-system/global-pull-secret-syncer-dx87q","kube-system/konnectivity-agent-v8crd","kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal"]
Jul 31 23:46:17.438588 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.438567    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:17.438695 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.438643    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:17.440812 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.440787    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.443110 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.443089    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.443900 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.443878    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.444012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.443884    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.444012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.443932    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqdpm\""
Jul 31 23:46:17.444012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.443970    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:46:17.445829 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445581    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.445829 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445684    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:46:17.445829 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445728    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:46:17.445829 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445770    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:46:17.446066 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445894    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:46:17.446066 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.445684    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.446177 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.446143    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xxtcd\""
Jul 31 23:46:17.446177 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.446172    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.447819 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.447640    2536 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:17.447819 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.447722    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.447979 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.447959    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.448391 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.448370    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4g4ll\""
Jul 31 23:46:17.448879 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.448691    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:46:17.448879 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.448740    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:46:17.448992 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.448932    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.449927 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.449909    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.450652 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.450632    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.450744 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.450676    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wbb5j\""
Jul 31 23:46:17.450918 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.450900    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.452037 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.452020    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.452267 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.452251    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.452350 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.452326    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:17.452496 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.452466    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.452610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.452594    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cbzwg\""
Jul 31 23:46:17.452679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.452596    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:46:17.454779 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.454762    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.456919 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.456900    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:46:17.457112 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.457093    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dczv6\""
Jul 31 23:46:17.457350 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.457339    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:46:17.457594 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.457564    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.459377 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459358    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.459616 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459597    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpfc\" (UniqueName: \"kubernetes.io/projected/e2d84a4a-4b9e-4903-8d41-f998c63622ee-kube-api-access-hdpfc\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.459713 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459629    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-kubelet\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.459713 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459669    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-cnibin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.459713 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459707    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-etc-kubernetes\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459753    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb504d-a872-4bfc-ad0c-976806a7def6-hosts-file\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459784    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb504d-a872-4bfc-ad0c-976806a7def6-tmp-dir\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459822    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ff29c5e-04e1-4d86-85e8-85a25196a97b-host-slash\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459850    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459883    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-kubelet\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.459913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459909    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-netns\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459931    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-systemd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.459954    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-config\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460006    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460038    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460041    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r9mhn\""
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460003    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460143    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-socket-dir-parent\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460175    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-k8s-cni-cncf-io\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460211    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-bin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460236    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-multus\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460266    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-systemd-units\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460288    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460311    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-node-log\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460333    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-netd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460355    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-hostroot\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460376    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-conf-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460391    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzls\" (UniqueName: \"kubernetes.io/projected/4d510acf-e79e-4e38-8faa-58281e5def06-kube-api-access-sbzls\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460412    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-env-overrides\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460472    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-system-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460504    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-cni-binary-copy\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460544    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-multus-daemon-config\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.460597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460591    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-multus-certs\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460620    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40464426-09cc-4746-93bc-2ec417a8b4c2-serviceca\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460642    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d58j\" (UniqueName: \"kubernetes.io/projected/40464426-09cc-4746-93bc-2ec417a8b4c2-kube-api-access-9d58j\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460694    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-log-socket\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460722    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460754    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-script-lib\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460807    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-os-release\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460852    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-netns\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460899    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r529\" (UniqueName: \"kubernetes.io/projected/dfcb504d-a872-4bfc-ad0c-976806a7def6-kube-api-access-5r529\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460924    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ff29c5e-04e1-4d86-85e8-85a25196a97b-iptables-alerter-script\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460949    2536 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460950    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5gl\" (UniqueName: \"kubernetes.io/projected/8ff29c5e-04e1-4d86-85e8-85a25196a97b-kube-api-access-cr5gl\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.460994    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-slash\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461018    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-bin\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461036    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461062    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40464426-09cc-4746-93bc-2ec417a8b4c2-host\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461090    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:17.461220 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461112    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-var-lib-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.462062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461136    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-etc-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.462062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461159    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-ovn\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.462062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461465    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:46:17.462062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461534    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:46:17.462062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.461533    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9v2lp\""
Jul 31 23:46:17.462299 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.462086    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.462299 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.462152    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:17.464510 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.464374    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.468733 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.468710    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:46:17.468733 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.468726    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:46:17.468887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.468720    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s2dww\""
Jul 31 23:46:17.468887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.468819    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:46:17.486453 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.486427    2536 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6x84v"
Jul 31 23:46:17.494965 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.494939    2536 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6x84v"
Jul 31 23:46:17.548818 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.548781    2536 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:46:17.561682 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561650    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-script-lib\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561690    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-os-release\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561716    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-netns\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561746    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-sys\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561771    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxmb\" (UniqueName: \"kubernetes.io/projected/d0eac260-1645-44ba-9ed6-549540cdef6d-kube-api-access-9xxmb\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561802    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5gl\" (UniqueName: \"kubernetes.io/projected/8ff29c5e-04e1-4d86-85e8-85a25196a97b-kube-api-access-cr5gl\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.561852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561827    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-slash\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561872    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561898    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40464426-09cc-4746-93bc-2ec417a8b4c2-host\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561956    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-slash\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.561963    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562019    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-os-release\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562050    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40464426-09cc-4746-93bc-2ec417a8b4c2-host\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562094    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-netns\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562139 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562093    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-registration-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562152    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-etc-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562181    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpfc\" (UniqueName: \"kubernetes.io/projected/e2d84a4a-4b9e-4903-8d41-f998c63622ee-kube-api-access-hdpfc\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562228    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-etc-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562278    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-system-cni-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562304    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-binary-copy\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562325    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-var-lib-kubelet\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562350    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-cnibin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562373    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562395    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-systemd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562407    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-cnibin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562443    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-systemd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562475    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-config\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562529    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562565    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-k8s-cni-cncf-io\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.562619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562592    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-bin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562630    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-cnibin\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562654    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-systemd\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562676    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-socket-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562699    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-k8s-cni-cncf-io\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562703    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-systemd-units\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562740    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562765    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-hostroot\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562770    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-bin\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562800    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562807    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-modprobe-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562833    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562852    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-systemd-units\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562856    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-lib-modules\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562884    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-hostroot\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562896    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-tmp\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562923    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-dbus\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.563300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562926    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-script-lib\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562957    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-env-overrides\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.562998    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-system-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563022    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-cni-binary-copy\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563064    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-system-cni-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563062    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-multus-certs\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563111    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e59a8686-fd5f-4ab2-a476-8379aed3312a-agent-certs\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563137    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r529\" (UniqueName: \"kubernetes.io/projected/dfcb504d-a872-4bfc-ad0c-976806a7def6-kube-api-access-5r529\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563163    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563113    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-run-multus-certs\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563188    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563213    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysconfig\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563248    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-conf\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563268    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-host\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563280    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovnkube-config\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563324    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e59a8686-fd5f-4ab2-a476-8379aed3312a-konnectivity-ca\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563344    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2d84a4a-4b9e-4903-8d41-f998c63622ee-env-overrides\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563385    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ff29c5e-04e1-4d86-85e8-85a25196a97b-iptables-alerter-script\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563441    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-bin\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563528    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-os-release\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563544    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-bin\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563596    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-run\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563627    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563671    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-var-lib-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563779    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-ovn\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563808    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-kubernetes\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563844    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-kubelet-config\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563872    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-kubelet\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563897    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-etc-kubernetes\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563919    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-var-lib-openvswitch\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563947    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-cni-binary-copy\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563976    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-etc-kubernetes\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563996    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-run-ovn\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.563995    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-kubelet\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.564872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564010    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb504d-a872-4bfc-ad0c-976806a7def6-hosts-file\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564016    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ff29c5e-04e1-4d86-85e8-85a25196a97b-iptables-alerter-script\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564035    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb504d-a872-4bfc-ad0c-976806a7def6-tmp-dir\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564089    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb504d-a872-4bfc-ad0c-976806a7def6-hosts-file\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564088    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-device-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564132    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ff29c5e-04e1-4d86-85e8-85a25196a97b-host-slash\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564161    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564195    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-kubelet\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564224    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ff29c5e-04e1-4d86-85e8-85a25196a97b-host-slash\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564226    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564254    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldtr2\" (UniqueName: \"kubernetes.io/projected/d6d3af55-3144-42d1-98f9-8b11c16bb667-kube-api-access-ldtr2\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564252    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-kubelet\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564279    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564308    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-netns\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564326    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb504d-a872-4bfc-ad0c-976806a7def6-tmp-dir\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564341    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-socket-dir-parent\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564366    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-multus\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.565645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564381    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-conf-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564389    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-netns\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564404    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-socket-dir-parent\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564416    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzls\" (UniqueName: \"kubernetes.io/projected/4d510acf-e79e-4e38-8faa-58281e5def06-kube-api-access-sbzls\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564431    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-host-var-lib-cni-multus\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564465    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snd95\" (UniqueName: \"kubernetes.io/projected/241aa0ea-8d3f-4d63-b482-12fb581f1810-kube-api-access-snd95\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564512    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564544    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-node-log\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564514    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d510acf-e79e-4e38-8faa-58281e5def06-multus-conf-dir\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564569    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-netd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564595    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40464426-09cc-4746-93bc-2ec417a8b4c2-serviceca\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564597    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-node-log\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564642    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-cni-netd\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564679    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d58j\" (UniqueName: \"kubernetes.io/projected/40464426-09cc-4746-93bc-2ec417a8b4c2-kube-api-access-9d58j\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564730    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564713    2536 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564755    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-tuned\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.566432 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564812    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564839    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-multus-daemon-config\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564863    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-sys-fs\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564887    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqdd\" (UniqueName: \"kubernetes.io/projected/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kube-api-access-ggqdd\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564927    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-log-socket\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.564989    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.565007    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40464426-09cc-4746-93bc-2ec417a8b4c2-serviceca\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.565027    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-log-socket\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.565079    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84a4a-4b9e-4903-8d41-f998c63622ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.567298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.565286    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d510acf-e79e-4e38-8faa-58281e5def06-multus-daemon-config\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.568622 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.568545    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2d84a4a-4b9e-4903-8d41-f998c63622ee-ovn-node-metrics-cert\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.571055 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.571032    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5gl\" (UniqueName: \"kubernetes.io/projected/8ff29c5e-04e1-4d86-85e8-85a25196a97b-kube-api-access-cr5gl\") pod \"iptables-alerter-x9kdv\" (UID: \"8ff29c5e-04e1-4d86-85e8-85a25196a97b\") " pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.571171 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.571128    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:17.571171 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.571154    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:17.571171 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.571168    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:17.571822 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.571259    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.071222117 +0000 UTC m=+3.036417740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:17.571822 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.571326    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r529\" (UniqueName: \"kubernetes.io/projected/dfcb504d-a872-4bfc-ad0c-976806a7def6-kube-api-access-5r529\") pod \"node-resolver-zlm46\" (UID: \"dfcb504d-a872-4bfc-ad0c-976806a7def6\") " pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.571822 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.571519    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpfc\" (UniqueName: \"kubernetes.io/projected/e2d84a4a-4b9e-4903-8d41-f998c63622ee-kube-api-access-hdpfc\") pod \"ovnkube-node-d9cnv\" (UID: \"e2d84a4a-4b9e-4903-8d41-f998c63622ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.573696 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.573674    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d58j\" (UniqueName: \"kubernetes.io/projected/40464426-09cc-4746-93bc-2ec417a8b4c2-kube-api-access-9d58j\") pod \"node-ca-bmzcw\" (UID: \"40464426-09cc-4746-93bc-2ec417a8b4c2\") " pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.573846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.573827    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzls\" (UniqueName: \"kubernetes.io/projected/4d510acf-e79e-4e38-8faa-58281e5def06-kube-api-access-sbzls\") pod \"multus-lv99k\" (UID: \"4d510acf-e79e-4e38-8faa-58281e5def06\") " pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.665536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665479    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-cnibin\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.665536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665538    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-systemd\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665564    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-socket-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665642    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-cnibin\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665655    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-systemd\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665692    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-modprobe-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665723    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-socket-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665728    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-lib-modules\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.665780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665749    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-tmp\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665804    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-dbus\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665835    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e59a8686-fd5f-4ab2-a476-8379aed3312a-agent-certs\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665863    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665868    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-modprobe-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665887    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-lib-modules\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665926    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-dbus\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.665979    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666005    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysconfig\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666032    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-conf\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666055    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-host\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666079    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e59a8686-fd5f-4ab2-a476-8379aed3312a-konnectivity-ca\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666080    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysconfig\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666117    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-os-release\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666140    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-run\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666157    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-host\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666180    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-kubernetes\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666204    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-kubelet-config\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666234    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-device-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666264    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666285    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-conf\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666288    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldtr2\" (UniqueName: \"kubernetes.io/projected/d6d3af55-3144-42d1-98f9-8b11c16bb667-kube-api-access-ldtr2\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666330    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666371    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snd95\" (UniqueName: \"kubernetes.io/projected/241aa0ea-8d3f-4d63-b482-12fb581f1810-kube-api-access-snd95\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666397    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666427    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666455    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-tuned\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666479    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.666599 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666476    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666523    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-sys-fs\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666550    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqdd\" (UniqueName: \"kubernetes.io/projected/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kube-api-access-ggqdd\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666581    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-sys\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.666601    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666604    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxmb\" (UniqueName: \"kubernetes.io/projected/d0eac260-1645-44ba-9ed6-549540cdef6d-kube-api-access-9xxmb\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666635    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-registration-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.666669    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.166648571 +0000 UTC m=+3.131844175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666669    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666696    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-system-cni-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666722    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-binary-copy\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666739    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666748    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-var-lib-kubelet\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666778    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.666897    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667023    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-registration-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.667375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667071    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-system-cni-dir\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667126    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e59a8686-fd5f-4ab2-a476-8379aed3312a-konnectivity-ca\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667176    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-sys-fs\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667205    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241aa0ea-8d3f-4d63-b482-12fb581f1810-os-release\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.667247    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667256    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-run\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:17.667292    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:18.167276821 +0000 UTC m=+3.132472441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667245    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667311    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-kubernetes\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667314    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-sys\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667321    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-kubelet-config\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667351    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-var-lib-kubelet\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667385    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-device-dir\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667436    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-sysctl-d\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.668117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.667555    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241aa0ea-8d3f-4d63-b482-12fb581f1810-cni-binary-copy\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.668823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.668506    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-tmp\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.669297 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.669144    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e59a8686-fd5f-4ab2-a476-8379aed3312a-agent-certs\") pod \"konnectivity-agent-v8crd\" (UID: \"e59a8686-fd5f-4ab2-a476-8379aed3312a\") " pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.669560 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.669542    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6d3af55-3144-42d1-98f9-8b11c16bb667-etc-tuned\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.674912 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.674885    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldtr2\" (UniqueName: \"kubernetes.io/projected/d6d3af55-3144-42d1-98f9-8b11c16bb667-kube-api-access-ldtr2\") pod \"tuned-6fnf7\" (UID: \"d6d3af55-3144-42d1-98f9-8b11c16bb667\") " pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.675013 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.674948    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqdd\" (UniqueName: \"kubernetes.io/projected/bd9b3a6c-59a6-4c75-b510-21a279bf6f05-kube-api-access-ggqdd\") pod \"aws-ebs-csi-driver-node-sx6bz\" (UID: \"bd9b3a6c-59a6-4c75-b510-21a279bf6f05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.675543 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.675523    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snd95\" (UniqueName: \"kubernetes.io/projected/241aa0ea-8d3f-4d63-b482-12fb581f1810-kube-api-access-snd95\") pod \"multus-additional-cni-plugins-n49r5\" (UID: \"241aa0ea-8d3f-4d63-b482-12fb581f1810\") " pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.675543 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.675538    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxmb\" (UniqueName: \"kubernetes.io/projected/d0eac260-1645-44ba-9ed6-549540cdef6d-kube-api-access-9xxmb\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:17.753113 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.753029    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9kdv"
Jul 31 23:46:17.761132 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.761109    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:17.770277 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.770254    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lv99k"
Jul 31 23:46:17.776997 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.776965    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlm46"
Jul 31 23:46:17.783704 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.783679    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmzcw"
Jul 31 23:46:17.790368 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.790343    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n49r5"
Jul 31 23:46:17.796997 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.796978    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6fnf7"
Jul 31 23:46:17.803652 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.803634    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:17.810218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.810196    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz"
Jul 31 23:46:17.876306 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:17.876275    2536 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:18.169698 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.169608    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:18.169698 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.169659    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:18.169889 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.169709    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:18.169889 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169815    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:18.169889 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169847    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:18.169889 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169861    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:18.170061 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169821    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.170061 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169926    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.16990581 +0000 UTC m=+4.135101429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:18.170061 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169957    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.169941523 +0000 UTC m=+4.135137128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.170061 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.169826    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.170061 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.170004    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.169992528 +0000 UTC m=+4.135188141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.366243 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.366173    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d510acf_e79e_4e38_8faa_58281e5def06.slice/crio-52c28c46aa6e987edced2e9bb4fe46ad859329c89acd8027fad00979b50da7d6 WatchSource:0}: Error finding container 52c28c46aa6e987edced2e9bb4fe46ad859329c89acd8027fad00979b50da7d6: Status 404 returned error can't find the container with id 52c28c46aa6e987edced2e9bb4fe46ad859329c89acd8027fad00979b50da7d6
Jul 31 23:46:18.374176 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.374061    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59a8686_fd5f_4ab2_a476_8379aed3312a.slice/crio-ab4fb0acdde8a8e18f0c765f64d365de1cf91dcfadb71f41ec8c02aec8425711 WatchSource:0}: Error finding container ab4fb0acdde8a8e18f0c765f64d365de1cf91dcfadb71f41ec8c02aec8425711: Status 404 returned error can't find the container with id ab4fb0acdde8a8e18f0c765f64d365de1cf91dcfadb71f41ec8c02aec8425711
Jul 31 23:46:18.374878 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.374852    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241aa0ea_8d3f_4d63_b482_12fb581f1810.slice/crio-14886e8c483d138847cf1b31042ce38f3701e6a6698048bd783681d100f94bcd WatchSource:0}: Error finding container 14886e8c483d138847cf1b31042ce38f3701e6a6698048bd783681d100f94bcd: Status 404 returned error can't find the container with id 14886e8c483d138847cf1b31042ce38f3701e6a6698048bd783681d100f94bcd
Jul 31 23:46:18.395325 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.395299    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d3af55_3144_42d1_98f9_8b11c16bb667.slice/crio-1584cfce18f1cbf1f09f32cd0a1c9a98470df09f26b39671179feda00fa4bfff WatchSource:0}: Error finding container 1584cfce18f1cbf1f09f32cd0a1c9a98470df09f26b39671179feda00fa4bfff: Status 404 returned error can't find the container with id 1584cfce18f1cbf1f09f32cd0a1c9a98470df09f26b39671179feda00fa4bfff
Jul 31 23:46:18.396120 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.396098    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9b3a6c_59a6_4c75_b510_21a279bf6f05.slice/crio-b5d00326a55d5f206f374a6d9e28588aecdfdcfc76a5f9238140a374008d651d WatchSource:0}: Error finding container b5d00326a55d5f206f374a6d9e28588aecdfdcfc76a5f9238140a374008d651d: Status 404 returned error can't find the container with id b5d00326a55d5f206f374a6d9e28588aecdfdcfc76a5f9238140a374008d651d
Jul 31 23:46:18.397665 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:18.397638    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff29c5e_04e1_4d86_85e8_85a25196a97b.slice/crio-3c6f60ffb2c11868e80cfe377daf3fad0a7f1914c4646492a06b80f895b6a326 WatchSource:0}: Error finding container 3c6f60ffb2c11868e80cfe377daf3fad0a7f1914c4646492a06b80f895b6a326: Status 404 returned error can't find the container with id 3c6f60ffb2c11868e80cfe377daf3fad0a7f1914c4646492a06b80f895b6a326
Jul 31 23:46:18.498792 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.498755    2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:17 +0000 UTC" deadline="2028-02-26 12:03:25.082933776 +0000 UTC"
Jul 31 23:46:18.498792 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.498786    2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13788h17m6.584151076s"
Jul 31 23:46:18.555327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.555146    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:18.555520 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.555146    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:18.555520 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.555462    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:18.555625 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:18.555527    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:18.564775 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.564735    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal" event={"ID":"8e48bbd57a2cb073c5f8cf46fee7320a","Type":"ContainerStarted","Data":"c346e820311c2c039e3e93405d0886c079b99fed710c4f904885b019835654a9"}
Jul 31 23:46:18.565951 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.565928    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9kdv" event={"ID":"8ff29c5e-04e1-4d86-85e8-85a25196a97b","Type":"ContainerStarted","Data":"3c6f60ffb2c11868e80cfe377daf3fad0a7f1914c4646492a06b80f895b6a326"}
Jul 31 23:46:18.567065 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.567044    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6fnf7" event={"ID":"d6d3af55-3144-42d1-98f9-8b11c16bb667","Type":"ContainerStarted","Data":"1584cfce18f1cbf1f09f32cd0a1c9a98470df09f26b39671179feda00fa4bfff"}
Jul 31 23:46:18.568180 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.568156    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerStarted","Data":"14886e8c483d138847cf1b31042ce38f3701e6a6698048bd783681d100f94bcd"}
Jul 31 23:46:18.569455 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.569399    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmzcw" event={"ID":"40464426-09cc-4746-93bc-2ec417a8b4c2","Type":"ContainerStarted","Data":"750b8518393c71e65845de9c80fcdd1dc7b9e8d8bba607e92eaffc853cac634a"}
Jul 31 23:46:18.570620 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.570597    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lv99k" event={"ID":"4d510acf-e79e-4e38-8faa-58281e5def06","Type":"ContainerStarted","Data":"52c28c46aa6e987edced2e9bb4fe46ad859329c89acd8027fad00979b50da7d6"}
Jul 31 23:46:18.571763 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.571742    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz" event={"ID":"bd9b3a6c-59a6-4c75-b510-21a279bf6f05","Type":"ContainerStarted","Data":"b5d00326a55d5f206f374a6d9e28588aecdfdcfc76a5f9238140a374008d651d"}
Jul 31 23:46:18.572839 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.572813    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8crd" event={"ID":"e59a8686-fd5f-4ab2-a476-8379aed3312a","Type":"ContainerStarted","Data":"ab4fb0acdde8a8e18f0c765f64d365de1cf91dcfadb71f41ec8c02aec8425711"}
Jul 31 23:46:18.573820 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.573803    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlm46" event={"ID":"dfcb504d-a872-4bfc-ad0c-976806a7def6","Type":"ContainerStarted","Data":"039dabd8efba3ce2d1781b613e6bad1bcd0ec16e4f4c5410b9e3a2de9110d8a6"}
Jul 31 23:46:18.574973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.574951    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"d62ed9fe72d083a798c02c68a1a3f8ee68f76d3dd90863e8c170e7add2286370"}
Jul 31 23:46:18.584698 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:18.584581    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-129.ec2.internal" podStartSLOduration=1.584562745 podStartE2EDuration="1.584562745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:18.580996443 +0000 UTC m=+3.546192062" watchObservedRunningTime="2026-07-31 23:46:18.584562745 +0000 UTC m=+3.549758368"
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.177407    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.177473    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.177530    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177593    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177662    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177669    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:21.177648715 +0000 UTC m=+6.142844348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177713    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:21.177696528 +0000 UTC m=+6.142892148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177811    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177828    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177841    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.177901 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.177877    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:21.177865331 +0000 UTC m=+6.143060946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.499885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.499757    2536 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:17 +0000 UTC" deadline="2028-01-04 01:37:31.448058001 +0000 UTC"
Jul 31 23:46:19.499885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.499822    2536 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12505h51m11.948240567s"
Jul 31 23:46:19.558782 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.558252    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:19.558782 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:19.558389    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:19.590023 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.589966    2536 generic.go:356] "Generic (PLEG): container finished" podID="9f4dc6b340f9875c6e96d1cfc5fb33dd" containerID="8258aeb1cf3d2d74e329a719d0c835733e56e728bd57e501ee19f1cbef464fba" exitCode=0
Jul 31 23:46:19.591202 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:19.591052    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal" event={"ID":"9f4dc6b340f9875c6e96d1cfc5fb33dd","Type":"ContainerDied","Data":"8258aeb1cf3d2d74e329a719d0c835733e56e728bd57e501ee19f1cbef464fba"}
Jul 31 23:46:20.555600 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:20.555568    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:20.556094 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:20.555715    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:20.556169 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:20.556148    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:20.556266 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:20.556242    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:20.613004 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:20.612919    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal" event={"ID":"9f4dc6b340f9875c6e96d1cfc5fb33dd","Type":"ContainerStarted","Data":"58dbaa75e82d1d887ac175e604c830fe2549bfc1a683e1cb970db91fbfcfc708"}
Jul 31 23:46:20.628752 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:20.628693    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-129.ec2.internal" podStartSLOduration=3.6286723370000002 podStartE2EDuration="3.628672337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.628507774 +0000 UTC m=+5.593703395" watchObservedRunningTime="2026-07-31 23:46:20.628672337 +0000 UTC m=+5.593867961"
Jul 31 23:46:21.205131 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:21.205087    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:21.205335 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:21.205170    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:21.205335 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:21.205209    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:21.205455 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205334    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:21.205455 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205399    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:25.205379812 +0000 UTC m=+10.170575414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205802    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205855    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:25.205839844 +0000 UTC m=+10.171035453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205932    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205946    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205958    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:21.206055 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.205988    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:25.205978408 +0000 UTC m=+10.171174015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:21.556185 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:21.555590    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:21.556185 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:21.555723    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:22.555554 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:22.555519    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:22.555759 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:22.555525    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:22.555759 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:22.555653    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:22.555876 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:22.555766    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:23.555799 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:23.555765    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:23.556294 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:23.555903    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:24.555349 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:24.555253    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:24.555697 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:24.555451    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:24.555697 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:24.555562    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:24.555861 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:24.555690    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:25.239265 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:25.239217    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:25.239544 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:25.239294    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:25.239544 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:25.239354    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:25.239544 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.239511    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:25.239685 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.239609    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:33.23958506 +0000 UTC m=+18.204780685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:25.240002 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.239978    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:25.240175 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.240039    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:25.240175 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.240052    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:33.24003514 +0000 UTC m=+18.205230754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:25.240175 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.240062    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:25.240175 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.240076    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:25.240175 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.240131    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:33.240116132 +0000 UTC m=+18.205311761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:25.556720 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:25.556238    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:25.556720 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:25.556353    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:26.555950 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:26.555907    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:26.555950 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:26.555932    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:26.556221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:26.556065    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:26.556281 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:26.556221    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:27.555339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:27.555302    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:27.555877 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:27.555421    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:28.555478 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:28.555437    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:28.555959 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:28.555502    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:28.555959 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:28.555604    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:28.555959 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:28.555757    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:29.556149 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:29.556112    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:29.556681 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:29.556258    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:30.555768 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:30.555726    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:30.555938 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:30.555726    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:30.555938 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:30.555896    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:30.556037 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:30.556008    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:31.555935 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:31.555894    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:31.556403 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:31.556030    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:32.555850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:32.555811    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:32.556067 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:32.555812    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:32.556067 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:32.555933    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:32.556067 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:32.556022    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:33.296823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:33.296784    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:33.297025 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:33.296850    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:33.297025 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:33.296901    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:33.297025 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.296961    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:33.297025 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297010    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297010    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297068    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297082    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297046    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.2970249 +0000 UTC m=+34.262220512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297126    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.297114214 +0000 UTC m=+34.262309819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:33.297229 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.297137    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.2971307 +0000 UTC m=+34.262326305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:33.556073 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:33.555955    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:33.556553 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:33.556145    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:34.555082 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:34.555046    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:34.555263 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:34.555180    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:34.555263 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:34.555246    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:34.555389 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:34.555365    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:35.555680 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:35.555648    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:35.556021 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:35.555738    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:36.555500 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:36.555452    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:36.555672 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:36.555606    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:36.555925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:36.555455    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:36.555925 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:36.555889    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:37.555330 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.555032    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:37.555480 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:37.555446    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:37.649991 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.649955    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6fnf7" event={"ID":"d6d3af55-3144-42d1-98f9-8b11c16bb667","Type":"ContainerStarted","Data":"0552380f44e0485374995af7bfff582e9cd5adbeefc7d027de01728f7e765e69"}
Jul 31 23:46:37.651692 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.651660    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="3e7d3fba9132aa32c5a5c264317eb46951e9b6babf4e5ba44b196c9475918522" exitCode=0
Jul 31 23:46:37.651836 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.651739    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"3e7d3fba9132aa32c5a5c264317eb46951e9b6babf4e5ba44b196c9475918522"}
Jul 31 23:46:37.654323 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.653447    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmzcw" event={"ID":"40464426-09cc-4746-93bc-2ec417a8b4c2","Type":"ContainerStarted","Data":"9b29264c76553cf4a69d7b07b10212f0fa0f858d7a699d6650f76fe2d4f512b0"}
Jul 31 23:46:37.659001 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.658734    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lv99k" event={"ID":"4d510acf-e79e-4e38-8faa-58281e5def06","Type":"ContainerStarted","Data":"a63463711adffd8fec5a51e53fd56c19c7c351b1132fae5545e535d3a4cb7fb7"}
Jul 31 23:46:37.660502 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.660466    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz" event={"ID":"bd9b3a6c-59a6-4c75-b510-21a279bf6f05","Type":"ContainerStarted","Data":"8e5aa58bf97fab7a61ec2736fbc1f7e6a8d18554aca59c3f41bacf6d499d9e8f"}
Jul 31 23:46:37.661998 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.661972    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8crd" event={"ID":"e59a8686-fd5f-4ab2-a476-8379aed3312a","Type":"ContainerStarted","Data":"d19eaa82173d94e4bd6e64e8b2a3d161b6b1da6d8a8fa31de282129883cafc6c"}
Jul 31 23:46:37.663475 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.663446    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlm46" event={"ID":"dfcb504d-a872-4bfc-ad0c-976806a7def6","Type":"ContainerStarted","Data":"85dc1d077a4a7ef6aa91f6d695ddbe2d2e230c7906f8913133eb63edc8414a8e"}
Jul 31 23:46:37.670055 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.670017    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6fnf7" podStartSLOduration=3.723620973 podStartE2EDuration="22.670005252s" totalImagesPullingTime="18.946384279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.669612734 +0000 UTC m=+22.634808357" watchObservedRunningTime="2026-07-31 23:46:37.670005252 +0000 UTC m=+22.635200872"
Jul 31 23:46:37.686621 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.686566    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zlm46" podStartSLOduration=3.735438679 podStartE2EDuration="22.686544625s" totalImagesPullingTime="18.951105946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.685648309 +0000 UTC m=+22.650843932" watchObservedRunningTime="2026-07-31 23:46:37.686544625 +0000 UTC m=+22.651740272"
Jul 31 23:46:37.709832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.709787    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmzcw" podStartSLOduration=3.75804676 podStartE2EDuration="22.709771287s" totalImagesPullingTime="18.951724527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.709391745 +0000 UTC m=+22.674587368" watchObservedRunningTime="2026-07-31 23:46:37.709771287 +0000 UTC m=+22.674966910"
Jul 31 23:46:37.734236 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.734188    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v8crd" podStartSLOduration=3.196980464 podStartE2EDuration="21.73417382s" totalImagesPullingTime="18.537193356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.733776851 +0000 UTC m=+22.698972475" watchObservedRunningTime="2026-07-31 23:46:37.73417382 +0000 UTC m=+22.699369772"
Jul 31 23:46:37.759417 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:37.759361    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lv99k" podStartSLOduration=3.784231808 podStartE2EDuration="22.759342792s" totalImagesPullingTime="18.975110984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:37.759213805 +0000 UTC m=+22.724409428" watchObservedRunningTime="2026-07-31 23:46:37.759342792 +0000 UTC m=+22.724538415"
Jul 31 23:46:38.555839 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.555802    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:38.556010 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.555853    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:38.556010 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:38.555935    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:38.556148 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:38.556122    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:38.676353 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676256    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"05fdbe8042fc871d31fb3c4577a5fff5a4bc35457517d4ac561e913a4b330f31"}
Jul 31 23:46:38.677097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676359    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"f95149e385cc5ad5411d022d2dff7854571d121a346583a2c08a6324665d6962"}
Jul 31 23:46:38.677097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676408    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"18207f1555fbf4396326027f127fdf811c3baece11ed039ea6f2295881da9a7a"}
Jul 31 23:46:38.677097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676454    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"c68593028a2b3c45024906bf60a2df5cd14fc07d28a0f7562d1d60346b3bb6ab"}
Jul 31 23:46:38.677097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676503    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"7de2c55cf2ffac906b00f142e8d57e805e9ee0faa033b687c2dafb9c5ffd616d"}
Jul 31 23:46:38.677097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.676536    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"d77f76e56aaa4ed428db261a60416538a537fe43b98769fe61d23e65072fb3ee"}
Jul 31 23:46:38.936647 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:38.936578    2536 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:39.177804 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.177770    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:39.178619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.178596    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:39.510889 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.510781    2536 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:38.936604762Z","UUID":"694460dc-07bf-46e7-93c3-aac27a14d093","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:39.513452 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.513247    2536 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:39.513452 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.513299    2536 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:39.556060 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.556020    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:39.556249 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:39.556136    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:39.681519 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.681469    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz" event={"ID":"bd9b3a6c-59a6-4c75-b510-21a279bf6f05","Type":"ContainerStarted","Data":"6147a53a5dca4e345ef13e821ae66749c8f546ebbd20084513d60c2fca0ffbaf"}
Jul 31 23:46:39.683795 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.683731    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9kdv" event={"ID":"8ff29c5e-04e1-4d86-85e8-85a25196a97b","Type":"ContainerStarted","Data":"1d7ba5fa92e4a106639a6f65ccbe7a5370cd2ed5eb2f12ee85c912faba4c57e0"}
Jul 31 23:46:39.701239 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:39.701155    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x9kdv" podStartSLOduration=5.751935077 podStartE2EDuration="24.701133141s" totalImagesPullingTime="18.949198064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:39.700656635 +0000 UTC m=+24.665852259" watchObservedRunningTime="2026-07-31 23:46:39.701133141 +0000 UTC m=+24.666328763"
Jul 31 23:46:40.389005 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.388965    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:40.389580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.389557    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v8crd"
Jul 31 23:46:40.555639 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.555603    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:40.555825 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.555607    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:40.555825 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:40.555738    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:40.555942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:40.555834    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:40.689080 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.688993    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz" event={"ID":"bd9b3a6c-59a6-4c75-b510-21a279bf6f05","Type":"ContainerStarted","Data":"d95551d9cbc8b2e937025389933125932b6b90e3305847e62d30f61154714364"}
Jul 31 23:46:40.707546 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:40.707478    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx6bz" podStartSLOduration=3.52246427 podStartE2EDuration="24.707460762s" totalImagesPullingTime="21.184996492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:40.707173329 +0000 UTC m=+25.672368952" watchObservedRunningTime="2026-07-31 23:46:40.707460762 +0000 UTC m=+25.672656385"
Jul 31 23:46:41.555711 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:41.555676    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:41.555903 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:41.555800    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:42.555980 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:42.555797    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:42.556516 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:42.555857    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:42.556516 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:42.556053    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:42.556516 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:42.556136    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:42.698222 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:42.698183    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"8c5301e7259813996ff7e5b1c4773ab14c07683d65d7b2a312f13b057c704057"}
Jul 31 23:46:42.699957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:42.699930    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="515abf33d82ac915da04dfd566a82d7f37c17bb9f27db16fa88c7161b6fec014" exitCode=0
Jul 31 23:46:42.700087 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:42.699970    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"515abf33d82ac915da04dfd566a82d7f37c17bb9f27db16fa88c7161b6fec014"}
Jul 31 23:46:43.555910 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:43.555882    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:43.556031 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:43.556011    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:43.704849 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:43.704752    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="2d41fee16297bc866bc0b774f5b26b0c9772dc9695a5b8a4469804ef419b95ff" exitCode=0
Jul 31 23:46:43.704849 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:43.704820    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"2d41fee16297bc866bc0b774f5b26b0c9772dc9695a5b8a4469804ef419b95ff"}
Jul 31 23:46:44.555461 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.555272    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:44.555712 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.555335    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:44.555712 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:44.555561    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:44.555712 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:44.555701    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:44.709935 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.709891    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" event={"ID":"e2d84a4a-4b9e-4903-8d41-f998c63622ee","Type":"ContainerStarted","Data":"359e51fdd8c3aa67c78e6747b2f0caaf57dd22b51e0e6f26ddc642c9d6268133"}
Jul 31 23:46:44.710563 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.710269    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:44.710563 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.710351    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:44.710563 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.710391    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:44.726281 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.726256    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:44.726404 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.726355    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:46:44.757540 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:44.757471    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv" podStartSLOduration=10.550026114 podStartE2EDuration="29.757456556s" totalImagesPullingTime="19.207430442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:44.757276632 +0000 UTC m=+29.722472255" watchObservedRunningTime="2026-07-31 23:46:44.757456556 +0000 UTC m=+29.722652178"
Jul 31 23:46:45.556099 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:45.556068    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:45.556242 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:45.556164    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:45.714054 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:45.714017    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="4d70d9cd1c231d1923f80ef71df4e1ec659854bfe4bf18bf14b9f5d48fb13444" exitCode=0
Jul 31 23:46:45.714461 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:45.714117    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"4d70d9cd1c231d1923f80ef71df4e1ec659854bfe4bf18bf14b9f5d48fb13444"}
Jul 31 23:46:46.171332 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.171295    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dx87q"]
Jul 31 23:46:46.171514 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.171470    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:46.171638 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:46.171595    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:46.173864 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.173836    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8kmsg"]
Jul 31 23:46:46.174125 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.174108    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:46.174334 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:46.174312    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:46.175532 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.175498    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq5g7"]
Jul 31 23:46:46.175673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:46.175657    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:46.175815 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:46.175789    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:47.555325 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:47.555104    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:47.555846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:47.555137    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:47.555846 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:47.555450    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:47.555846 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:47.555528    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:48.555939 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:48.555902    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:48.556414 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:48.556050    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx87q" podUID="f5475fe4-6ba2-4652-84f5-0f5a887b4a0e"
Jul 31 23:46:49.322887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:49.322847    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:49.322887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:49.322903    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:49.322935    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323021    2536 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323043    2536 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323075    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323095    2536 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323103    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret podName:f5475fe4-6ba2-4652-84f5-0f5a887b4a0e nodeName:}" failed. No retries permitted until 2026-07-31 23:47:21.323079895 +0000 UTC m=+66.288275520 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret") pod "global-pull-secret-syncer-dx87q" (UID: "f5475fe4-6ba2-4652-84f5-0f5a887b4a0e") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323108    2536 projected.go:196] Error preparing data for projected volume kube-api-access-5wptl for pod openshift-network-diagnostics/network-check-target-8kmsg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323124    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs podName:d0eac260-1645-44ba-9ed6-549540cdef6d nodeName:}" failed. No retries permitted until 2026-07-31 23:47:21.323113277 +0000 UTC m=+66.288308894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs") pod "network-metrics-daemon-zq5g7" (UID: "d0eac260-1645-44ba-9ed6-549540cdef6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:49.323138 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.323144    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl podName:0fc4f897-6876-4f7e-a1d5-f897546761fb nodeName:}" failed. No retries permitted until 2026-07-31 23:47:21.323134346 +0000 UTC m=+66.288329951 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wptl" (UniqueName: "kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl") pod "network-check-target-8kmsg" (UID: "0fc4f897-6876-4f7e-a1d5-f897546761fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:49.555445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:49.555409    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:49.555644 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.555558    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kmsg" podUID="0fc4f897-6876-4f7e-a1d5-f897546761fb"
Jul 31 23:46:49.555723 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:49.555641    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:49.555795 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:49.555771    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq5g7" podUID="d0eac260-1645-44ba-9ed6-549540cdef6d"
Jul 31 23:46:50.361504 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.361461    2536 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-129.ec2.internal" event="NodeReady"
Jul 31 23:46:50.362045 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.361595    2536 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:50.395703 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.395665    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:46:50.398961 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.398934    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"]
Jul 31 23:46:50.399107 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.399065    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.401126 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.401098    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 23:46:50.401504 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.401471    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 23:46:50.401700 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.401676    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 23:46:50.401777 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.401685    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxfp4\""
Jul 31 23:46:50.402208 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.402193    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"]
Jul 31 23:46:50.402293 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.402275    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.404563 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.404544    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 31 23:46:50.404969 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.404950    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.405419 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.405397    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zdghk"]
Jul 31 23:46:50.405552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.405534    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.406117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.405403    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.406308 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.406287    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rbrbb\""
Jul 31 23:46:50.407807 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.407785    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 23:46:50.408346 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408329    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.408524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408507    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8m6jg\""
Jul 31 23:46:50.408524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408519    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 31 23:46:50.408681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408533    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 31 23:46:50.408681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408514    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.408681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.408623    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.411362 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411142    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 23:46:50.411362 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411215    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.411362 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411244    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.411362 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411296    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 23:46:50.411687 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411464    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"]
Jul 31 23:46:50.411687 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.411645    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jnbbh\""
Jul 31 23:46:50.414717 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.414695    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"]
Jul 31 23:46:50.414850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.414825    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.417837 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.417813    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-g2zqs"]
Jul 31 23:46:50.419111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.418893    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 23:46:50.419248 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.419229    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.420208 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.420077    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 23:46:50.420208 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.420107    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.421050 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.420738    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.422826 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.421282    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"]
Jul 31 23:46:50.422826 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.421326    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:46:50.422826 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.421540    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.422826 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.421753    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vddmg\""
Jul 31 23:46:50.425300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.425145    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"]
Jul 31 23:46:50.425695 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.425675    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.425850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.425764    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 23:46:50.425984 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.425884    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.426420 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.426395    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 23:46:50.426821 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.426709    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c4ddm\""
Jul 31 23:46:50.427002 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.426985    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 23:46:50.428185 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.427823    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.428539 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.428338    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 23:46:50.428539 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.428389    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 23:46:50.428695 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.428633    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br2vn\""
Jul 31 23:46:50.428695 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.428643    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"]
Jul 31 23:46:50.428792 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.428758    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.431012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.430954    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"]
Jul 31 23:46:50.434113 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.434080    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 23:46:50.434424 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.434395    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-g2zqs"]
Jul 31 23:46:50.435213 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.435189    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cgcbc"]
Jul 31 23:46:50.438915 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.438870    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"]
Jul 31 23:46:50.439047 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.438974    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.441411 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.441390    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:50.441411 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.441403    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:50.441592 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.441397    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rs996\""
Jul 31 23:46:50.442264 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.442245    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c4768f676-zrpn9"]
Jul 31 23:46:50.442382 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.442367    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"
Jul 31 23:46:50.444780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.444749    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-whpwl\""
Jul 31 23:46:50.444890 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.444785    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.445072 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.445057    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.445238 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.445212    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"]
Jul 31 23:46:50.445404 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.445351    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.447800 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.447614    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.448216 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448197    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 31 23:46:50.448399 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448378    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"]
Jul 31 23:46:50.448478 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448228    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 31 23:46:50.448608 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448585    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dkrqj\""
Jul 31 23:46:50.448742 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448724    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 31 23:46:50.448859 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.448838    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 31 23:46:50.449410 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.449135    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.449410 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.449193    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.452956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.452013    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 31 23:46:50.452956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.452516    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 31 23:46:50.452956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.452577    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dr4vj"]
Jul 31 23:46:50.452956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.452741    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w7zwr\""
Jul 31 23:46:50.455658 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.453610    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.458120    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zdghk"]
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.458187    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"]
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.458213    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr4vj"]
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.458354    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"]
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.458470    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.459270    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.460159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.459356    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5vd6d\""
Jul 31 23:46:50.460572 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.460208    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.461593 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.461517    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:50.461896 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.461766    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:50.461896 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.461849    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m6h4p\""
Jul 31 23:46:50.461896 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.461860    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:50.462561 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.462541    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgcbc"]
Jul 31 23:46:50.463879 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.463858    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c4768f676-zrpn9"]
Jul 31 23:46:50.465637 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.464639    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"]
Jul 31 23:46:50.532001 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.531961    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-config\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.532001 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.531999    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-stats-auth\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532033    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59b6t\" (UniqueName: \"kubernetes.io/projected/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-kube-api-access-59b6t\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532095    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532116    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532137    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rwb\" (UniqueName: \"kubernetes.io/projected/a39d93a8-1777-4329-ab31-24a680418b44-kube-api-access-b6rwb\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532156    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad66796-3dd4-48e5-856a-d7deaf771563-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532173    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzd9r\" (UniqueName: \"kubernetes.io/projected/1fe52f27-2747-45aa-a384-14548caf51b6-kube-api-access-fzd9r\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532188    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-default-certificate\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.532226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532203    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532242    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532272    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532315    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532329    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532346    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62cc\" (UniqueName: \"kubernetes.io/projected/2ad66796-3dd4-48e5-856a-d7deaf771563-kube-api-access-w62cc\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532365    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcf6\" (UniqueName: \"kubernetes.io/projected/134b2375-9a90-46ad-9980-5c5467e1a381-kube-api-access-ndcf6\") pod \"network-check-source-65bdbbff45-5dmjq\" (UID: \"134b2375-9a90-46ad-9980-5c5467e1a381\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532382    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-service-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532481    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532544    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532572    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e6b3eb5-7d66-4655-b38f-863883402a84-serving-cert\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532591    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmjk\" (UniqueName: \"kubernetes.io/projected/b87aa9ef-45e9-4918-a2e5-96188fc8f544-kube-api-access-8bmjk\") pod \"volume-data-source-validator-5b8776669-2kqrw\" (UID: \"b87aa9ef-45e9-4918-a2e5-96188fc8f544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532607    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-serving-cert\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532636    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rck8p\" (UniqueName: \"kubernetes.io/projected/7e6b3eb5-7d66-4655-b38f-863883402a84-kube-api-access-rck8p\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532659    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532674    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-tmp\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532701    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvtf\" (UniqueName: \"kubernetes.io/projected/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-kube-api-access-mhvtf\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532733    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-tmp-dir\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532762    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe52f27-2747-45aa-a384-14548caf51b6-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532829    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-config-volume\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532863    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.532925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532913    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-trusted-ca\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532932    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72q8n\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532967    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532982    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dn9\" (UniqueName: \"kubernetes.io/projected/75c48d66-57f1-495e-97d3-8f735cf965fe-kube-api-access-88dn9\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.532999    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe52f27-2747-45aa-a384-14548caf51b6-config\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533012    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-trusted-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533030    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsh7x\" (UniqueName: \"kubernetes.io/projected/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-kube-api-access-jsh7x\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533046    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad66796-3dd4-48e5-856a-d7deaf771563-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533077    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/316ac0b7-e394-405e-b5eb-3c5575a64dbe-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533102    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533123    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c48d66-57f1-495e-97d3-8f735cf965fe-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.533433 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.533146    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-snapshots\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.555304 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.555271    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:46:50.557395 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.557372    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:50.602157 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.602112    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.634163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634061    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsh7x\" (UniqueName: \"kubernetes.io/projected/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-kube-api-access-jsh7x\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.634163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634110    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad66796-3dd4-48e5-856a-d7deaf771563-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.634163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634139    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/316ac0b7-e394-405e-b5eb-3c5575a64dbe-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634259    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634296    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c48d66-57f1-495e-97d3-8f735cf965fe-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634320    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-snapshots\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634342    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-config\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634370    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-stats-auth\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.634413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634410    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59b6t\" (UniqueName: \"kubernetes.io/projected/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-kube-api-access-59b6t\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.634439    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.134419114 +0000 UTC m=+36.099614750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634508    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634536    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634597    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rwb\" (UniqueName: \"kubernetes.io/projected/a39d93a8-1777-4329-ab31-24a680418b44-kube-api-access-b6rwb\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634622    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad66796-3dd4-48e5-856a-d7deaf771563-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634648    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-cert\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634671    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd9r\" (UniqueName: \"kubernetes.io/projected/1fe52f27-2747-45aa-a384-14548caf51b6-kube-api-access-fzd9r\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634695    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-default-certificate\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634718    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.634748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634746    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634770    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6llc\" (UniqueName: \"kubernetes.io/projected/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-kube-api-access-d6llc\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634799    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634857    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634888    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634918    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62cc\" (UniqueName: \"kubernetes.io/projected/2ad66796-3dd4-48e5-856a-d7deaf771563-kube-api-access-w62cc\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634946    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcf6\" (UniqueName: \"kubernetes.io/projected/134b2375-9a90-46ad-9980-5c5467e1a381-kube-api-access-ndcf6\") pod \"network-check-source-65bdbbff45-5dmjq\" (UID: \"134b2375-9a90-46ad-9980-5c5467e1a381\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.634972    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-service-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635001    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635016    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/316ac0b7-e394-405e-b5eb-3c5575a64dbe-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635023    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635054    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-config\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635061    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-snapshots\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635084    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e6b3eb5-7d66-4655-b38f-863883402a84-serving-cert\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635093    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:50.635246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635114    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmjk\" (UniqueName: \"kubernetes.io/projected/b87aa9ef-45e9-4918-a2e5-96188fc8f544-kube-api-access-8bmjk\") pod \"volume-data-source-validator-5b8776669-2kqrw\" (UID: \"b87aa9ef-45e9-4918-a2e5-96188fc8f544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635137    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.135123468 +0000 UTC m=+36.100319087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635147    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635160    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-serving-cert\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635169    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c48d66-57f1-495e-97d3-8f735cf965fe-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635189    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rck8p\" (UniqueName: \"kubernetes.io/projected/7e6b3eb5-7d66-4655-b38f-863883402a84-kube-api-access-rck8p\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635199    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.135184286 +0000 UTC m=+36.100379889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635249    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635321    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635377    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635431    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.135417287 +0000 UTC m=+36.100612890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635509    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635522    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635559    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.135546136 +0000 UTC m=+36.100741738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.635594    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.135584638 +0000 UTC m=+36.100780239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:50.635942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635674    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad66796-3dd4-48e5-856a-d7deaf771563-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.635930    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.636902    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-service-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637155    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-tmp\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637189    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvtf\" (UniqueName: \"kubernetes.io/projected/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-kube-api-access-mhvtf\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637249    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-tmp-dir\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637288    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe52f27-2747-45aa-a384-14548caf51b6-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.637505 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637329    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-config-volume\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.637846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637594    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-tmp\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.637846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637598    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.637950 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637878    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:50.638002 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.637951    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-tmp-dir\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.638002 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.637967    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:50.638110 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638034    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-trusted-ca\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.638110 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:50.638057    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.138039639 +0000 UTC m=+36.103235255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:50.638110 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638099    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72q8n\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.638258 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638139    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.638258 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638166    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dn9\" (UniqueName: \"kubernetes.io/projected/75c48d66-57f1-495e-97d3-8f735cf965fe-kube-api-access-88dn9\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.638258 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638203    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe52f27-2747-45aa-a384-14548caf51b6-config\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.638258 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.638229    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-trusted-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.639544 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.639518    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-config-volume\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.640261 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.640198    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-trusted-ca-bundle\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.640435 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.640400    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e6b3eb5-7d66-4655-b38f-863883402a84-trusted-ca\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.640805 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.640778    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.641360 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.641337    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.641619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.641597    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.641759 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.641735    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe52f27-2747-45aa-a384-14548caf51b6-config\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.642058 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.642034    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e6b3eb5-7d66-4655-b38f-863883402a84-serving-cert\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.642703 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.642677    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad66796-3dd4-48e5-856a-d7deaf771563-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.642894 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.642874    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe52f27-2747-45aa-a384-14548caf51b6-serving-cert\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.646802 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.646772    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmjk\" (UniqueName: \"kubernetes.io/projected/b87aa9ef-45e9-4918-a2e5-96188fc8f544-kube-api-access-8bmjk\") pod \"volume-data-source-validator-5b8776669-2kqrw\" (UID: \"b87aa9ef-45e9-4918-a2e5-96188fc8f544\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"
Jul 31 23:46:50.647230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.647197    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62cc\" (UniqueName: \"kubernetes.io/projected/2ad66796-3dd4-48e5-856a-d7deaf771563-kube-api-access-w62cc\") pod \"kube-storage-version-migrator-operator-7878f4b88c-vjzn8\" (UID: \"2ad66796-3dd4-48e5-856a-d7deaf771563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.647349 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.647252    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcf6\" (UniqueName: \"kubernetes.io/projected/134b2375-9a90-46ad-9980-5c5467e1a381-kube-api-access-ndcf6\") pod \"network-check-source-65bdbbff45-5dmjq\" (UID: \"134b2375-9a90-46ad-9980-5c5467e1a381\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"
Jul 31 23:46:50.647678 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.647652    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzd9r\" (UniqueName: \"kubernetes.io/projected/1fe52f27-2747-45aa-a384-14548caf51b6-kube-api-access-fzd9r\") pod \"service-ca-operator-784f97bcf6-wvjfj\" (UID: \"1fe52f27-2747-45aa-a384-14548caf51b6\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.647822 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.647798    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-default-certificate\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.648622 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.648597    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-serving-cert\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.649459 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.649430    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-stats-auth\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.651119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.650922    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.651119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651054    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rck8p\" (UniqueName: \"kubernetes.io/projected/7e6b3eb5-7d66-4655-b38f-863883402a84-kube-api-access-rck8p\") pod \"console-operator-5ccd6b445f-zdghk\" (UID: \"7e6b3eb5-7d66-4655-b38f-863883402a84\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.651119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651063    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rwb\" (UniqueName: \"kubernetes.io/projected/a39d93a8-1777-4329-ab31-24a680418b44-kube-api-access-b6rwb\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:50.651119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651066    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsh7x\" (UniqueName: \"kubernetes.io/projected/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-kube-api-access-jsh7x\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:50.651119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651065    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dn9\" (UniqueName: \"kubernetes.io/projected/75c48d66-57f1-495e-97d3-8f735cf965fe-kube-api-access-88dn9\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:50.651389 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651264    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59b6t\" (UniqueName: \"kubernetes.io/projected/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-kube-api-access-59b6t\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:50.651582 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.651559    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72q8n\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:50.652156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.652135    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvtf\" (UniqueName: \"kubernetes.io/projected/d7fd0abc-48ff-4224-8cf3-2bb4b369193a-kube-api-access-mhvtf\") pod \"insights-operator-77696685b-g2zqs\" (UID: \"d7fd0abc-48ff-4224-8cf3-2bb4b369193a\") " pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.691327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.691293    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.727928 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.727904    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"
Jul 31 23:46:50.730105 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.730076    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="46eb4b785cf11305a71ece407e007460094593353cf8721210d6f5f7e2f24ae0" exitCode=0
Jul 31 23:46:50.730243 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.730125    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"46eb4b785cf11305a71ece407e007460094593353cf8721210d6f5f7e2f24ae0"}
Jul 31 23:46:50.738642 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.738614    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-cert\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.738779 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.738658    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6llc\" (UniqueName: \"kubernetes.io/projected/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-kube-api-access-d6llc\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.743125 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.741517    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:50.743125 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.743011    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-cert\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.746736 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.746709    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6llc\" (UniqueName: \"kubernetes.io/projected/ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150-kube-api-access-d6llc\") pod \"ingress-canary-dr4vj\" (UID: \"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150\") " pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.749397 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.749369    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"
Jul 31 23:46:50.767062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.767020    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-g2zqs"
Jul 31 23:46:50.774081 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.774010    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr4vj"
Jul 31 23:46:50.784465 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.784252    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"
Jul 31 23:46:50.802150 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.802105    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"
Jul 31 23:46:50.958517 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:50.957830    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj"]
Jul 31 23:46:50.981806 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:50.980824    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe52f27_2747_45aa_a384_14548caf51b6.slice/crio-8a60dee84cb021c0ff9d0af17ed4586e46ce39991791f904ac81992e61962ec9 WatchSource:0}: Error finding container 8a60dee84cb021c0ff9d0af17ed4586e46ce39991791f904ac81992e61962ec9: Status 404 returned error can't find the container with id 8a60dee84cb021c0ff9d0af17ed4586e46ce39991791f904ac81992e61962ec9
Jul 31 23:46:51.017159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.017086    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zdghk"]
Jul 31 23:46:51.029590 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.029141    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8"]
Jul 31 23:46:51.029590 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.029216    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-g2zqs"]
Jul 31 23:46:51.072050 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.071968    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr4vj"]
Jul 31 23:46:51.077871 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.077837    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw"]
Jul 31 23:46:51.085532 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.085502    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.090020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.089992    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq"]
Jul 31 23:46:51.093548 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:51.093521    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87aa9ef_45e9_4918_a2e5_96188fc8f544.slice/crio-0dffcf32c9e485150d161a2248f6f509126dfde53ab2ab03bd416a9adba07693 WatchSource:0}: Error finding container 0dffcf32c9e485150d161a2248f6f509126dfde53ab2ab03bd416a9adba07693: Status 404 returned error can't find the container with id 0dffcf32c9e485150d161a2248f6f509126dfde53ab2ab03bd416a9adba07693
Jul 31 23:46:51.094420 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:51.094398    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad66796_3dd4_48e5_856a_d7deaf771563.slice/crio-953c7052739520b17fc029956ec9a98e32610d197dd913424a574c9f2ee38774 WatchSource:0}: Error finding container 953c7052739520b17fc029956ec9a98e32610d197dd913424a574c9f2ee38774: Status 404 returned error can't find the container with id 953c7052739520b17fc029956ec9a98e32610d197dd913424a574c9f2ee38774
Jul 31 23:46:51.095669 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:51.095636    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02ce1a_ff7b_4c13_9c1c_04aa47ddf150.slice/crio-5ca60dc82c00b8ea8f600806364d47a5164b1811ec105b8c930307bb2c83a62b WatchSource:0}: Error finding container 5ca60dc82c00b8ea8f600806364d47a5164b1811ec105b8c930307bb2c83a62b: Status 404 returned error can't find the container with id 5ca60dc82c00b8ea8f600806364d47a5164b1811ec105b8c930307bb2c83a62b
Jul 31 23:46:51.096603 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:51.096112    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134b2375_9a90_46ad_9980_5c5467e1a381.slice/crio-9b16ceede8fce2d51272d8a968604a360c1d90a96ea2780c5be2ddcd6d994c4f WatchSource:0}: Error finding container 9b16ceede8fce2d51272d8a968604a360c1d90a96ea2780c5be2ddcd6d994c4f: Status 404 returned error can't find the container with id 9b16ceede8fce2d51272d8a968604a360c1d90a96ea2780c5be2ddcd6d994c4f
Jul 31 23:46:51.099199 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:46:51.099060    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fd0abc_48ff_4224_8cf3_2bb4b369193a.slice/crio-7f025e622edeaf254e7180d6faa5961f70ba017448debf3674148e040a0f38ec WatchSource:0}: Error finding container 7f025e622edeaf254e7180d6faa5961f70ba017448debf3674148e040a0f38ec: Status 404 returned error can't find the container with id 7f025e622edeaf254e7180d6faa5961f70ba017448debf3674148e040a0f38ec
Jul 31 23:46:51.147549 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147428    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:51.147549 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147470    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147587    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147591    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147620    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147666    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147647258 +0000 UTC m=+37.112842876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147601    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147688    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147703    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:51.147739 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147734    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147716809 +0000 UTC m=+37.112912418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147756    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147777    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147754584 +0000 UTC m=+37.112950188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147811    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147800908 +0000 UTC m=+37.112996509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147854    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147899    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147922    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.147940    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.147958    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147947235 +0000 UTC m=+37.113142836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.148005    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.147995075 +0000 UTC m=+37.113190676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.148025    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:51.148095 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:51.148057    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.148048429 +0000 UTC m=+37.113244030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:51.186101 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.186056    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.556722 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.555974    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:46:51.556722 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.556658    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:46:51.559197 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.559006    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5t5tm\""
Jul 31 23:46:51.559960 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.559553    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v7qx8\""
Jul 31 23:46:51.559960 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.559808    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:51.736341 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.736303    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw" event={"ID":"b87aa9ef-45e9-4918-a2e5-96188fc8f544","Type":"ContainerStarted","Data":"0dffcf32c9e485150d161a2248f6f509126dfde53ab2ab03bd416a9adba07693"}
Jul 31 23:46:51.750953 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.740892    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8" event={"ID":"2ad66796-3dd4-48e5-856a-d7deaf771563","Type":"ContainerStarted","Data":"953c7052739520b17fc029956ec9a98e32610d197dd913424a574c9f2ee38774"}
Jul 31 23:46:51.750953 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.744981    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr4vj" event={"ID":"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150","Type":"ContainerStarted","Data":"5ca60dc82c00b8ea8f600806364d47a5164b1811ec105b8c930307bb2c83a62b"}
Jul 31 23:46:51.760709 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.760671    2536 generic.go:356] "Generic (PLEG): container finished" podID="241aa0ea-8d3f-4d63-b482-12fb581f1810" containerID="5b36949c7c9965037dc7d10fe34cdaaa32690a884491a9a87d3ce2d58e58aa3f" exitCode=0
Jul 31 23:46:51.760862 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.760773    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerDied","Data":"5b36949c7c9965037dc7d10fe34cdaaa32690a884491a9a87d3ce2d58e58aa3f"}
Jul 31 23:46:51.765731 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.765694    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-g2zqs" event={"ID":"d7fd0abc-48ff-4224-8cf3-2bb4b369193a","Type":"ContainerStarted","Data":"7f025e622edeaf254e7180d6faa5961f70ba017448debf3674148e040a0f38ec"}
Jul 31 23:46:51.774194 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.773606    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" event={"ID":"7e6b3eb5-7d66-4655-b38f-863883402a84","Type":"ContainerStarted","Data":"d49ebc3b1e96a656bf6f51fdc70a1beb38cc111de585ecd5c5a8c989322e7669"}
Jul 31 23:46:51.780072 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.780038    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj" event={"ID":"1fe52f27-2747-45aa-a384-14548caf51b6","Type":"ContainerStarted","Data":"8a60dee84cb021c0ff9d0af17ed4586e46ce39991791f904ac81992e61962ec9"}
Jul 31 23:46:51.782297 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:51.782266    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq" event={"ID":"134b2375-9a90-46ad-9980-5c5467e1a381","Type":"ContainerStarted","Data":"9b16ceede8fce2d51272d8a968604a360c1d90a96ea2780c5be2ddcd6d994c4f"}
Jul 31 23:46:52.158337 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158015    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:52.158337 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158319    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:52.158610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158359    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:52.158610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158391    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:52.158610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158465    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:52.158610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158534    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:52.158610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.158589    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:52.158855 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158747    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.158727934 +0000 UTC m=+39.123923552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:52.158855 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158834    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:52.158964 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158869    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.158857384 +0000 UTC m=+39.124053000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:52.158964 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158940    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:52.158964 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158951    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:46:52.159113 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.158981    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.158971215 +0000 UTC m=+39.124166833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:46:52.159113 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159042    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:52.159113 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159072    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.159062773 +0000 UTC m=+39.124258388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:52.159263 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159128    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:52.159263 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159157    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.159147992 +0000 UTC m=+39.124343606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:46:52.159263 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159213    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:52.159263 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159244    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.159234307 +0000 UTC m=+39.124429923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:46:52.159460 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159294    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:52.159460 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:52.159324    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.159313712 +0000 UTC m=+39.124509328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:52.803881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.802853    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n49r5" event={"ID":"241aa0ea-8d3f-4d63-b482-12fb581f1810","Type":"ContainerStarted","Data":"fc10ed4218b7430ad8328f2af1a873358e05a70836e64a47da369f5161696ef7"}
Jul 31 23:46:52.827291 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.827222    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n49r5" podStartSLOduration=8.73894509 podStartE2EDuration="37.827197107s" totalImagesPullingTime="29.088252017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:52.82473513 +0000 UTC m=+37.789930756" watchObservedRunningTime="2026-07-31 23:46:52.827197107 +0000 UTC m=+37.792392731"
Jul 31 23:46:52.950204 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:52.950164    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.016922 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:53.016886    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.180403 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180365    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:54.180403 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180408    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180432    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180460    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180548    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180565    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180573    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180598    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180632    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180650    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180629316 +0000 UTC m=+43.145824930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180584    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180701    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180703    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180564    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180720    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180673592 +0000 UTC m=+43.145869216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:54.180759    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180815    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180801431 +0000 UTC m=+43.145997032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:46:54.180957 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180831    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180821795 +0000 UTC m=+43.146017402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:46:54.181546 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180846    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180838169 +0000 UTC m=+43.146033825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:54.181546 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180858    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180851368 +0000 UTC m=+43.146046972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:54.181546 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:54.180880    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:58.180872409 +0000 UTC m=+43.146068010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:55.770647 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:55.770603    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.829501 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:55.829393    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.654573 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:56.654527    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.715504 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:56.715454    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.981160 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:56.981070    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.083161 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:57.083122    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.217286 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217250    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217313    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217340    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217373    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217405    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217412    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217423    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.217406504 +0000 UTC m=+51.182602131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217495    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217507    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217514    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217517    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217502    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.21747101 +0000 UTC m=+51.182666624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217592    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.217583621 +0000 UTC m=+51.182779225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217605    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.217599177 +0000 UTC m=+51.182794778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217617    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.217612311 +0000 UTC m=+51.182807912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:46:58.217705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217678    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:46:58.218195 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.217716    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:46:58.218195 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217791    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:46:58.218195 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217818    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.21780957 +0000 UTC m=+51.183005171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:46:58.218195 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217827    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:58.218195 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:58.217888    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:06.217873249 +0000 UTC m=+51.183068862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:46:58.826525 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.825872    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq" event={"ID":"134b2375-9a90-46ad-9980-5c5467e1a381","Type":"ContainerStarted","Data":"b27970239c5fab8c32c0c644f74105279e892c92519903978135ccb28536c9c0"}
Jul 31 23:46:58.828949 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.828502    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw" event={"ID":"b87aa9ef-45e9-4918-a2e5-96188fc8f544","Type":"ContainerStarted","Data":"8cf06f5640f2e9a488ad585a1f5078ba72a7250e56fd1f5b74982119147c4c67"}
Jul 31 23:46:58.832127 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.831686    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8" event={"ID":"2ad66796-3dd4-48e5-856a-d7deaf771563","Type":"ContainerStarted","Data":"6b78459469d8c53760b246a89961f3f3b1eb541d2299f58274a73016f6c950d9"}
Jul 31 23:46:58.835718 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.834566    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr4vj" event={"ID":"ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150","Type":"ContainerStarted","Data":"dc97fe035fb586b89f778b7fd31b972ec3588f279bb86ad8025501e0a208ccd3"}
Jul 31 23:46:58.835843 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.835794    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-g2zqs" event={"ID":"d7fd0abc-48ff-4224-8cf3-2bb4b369193a","Type":"ContainerStarted","Data":"793f3fbe0bf48449245a9a32f65d6aa327b99161d7593c4ac6e25d564be6d04a"}
Jul 31 23:46:58.837351 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.837330    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" event={"ID":"7e6b3eb5-7d66-4655-b38f-863883402a84","Type":"ContainerStarted","Data":"09bbd2d3df9c25dccdb9625503ba8ef82c8aad4356e369868a9aceeea3078109"}
Jul 31 23:46:58.837762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.837723    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:46:58.839207 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.839175    2536 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-zdghk container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Jul 31 23:46:58.839310 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.839220    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" podUID="7e6b3eb5-7d66-4655-b38f-863883402a84" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Jul 31 23:46:58.839568 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.839529    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj" event={"ID":"1fe52f27-2747-45aa-a384-14548caf51b6","Type":"ContainerStarted","Data":"8fa66c6978b8a9565417ae707bbd5c42ce29bdd8d7a1614d9e80507b2d11f37a"}
Jul 31 23:46:58.860347 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.860287    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8" podStartSLOduration=29.475754284 podStartE2EDuration="36.860266919s" totalImagesPullingTime="7.384512635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.859822729 +0000 UTC m=+43.825018354" watchObservedRunningTime="2026-07-31 23:46:58.860266919 +0000 UTC m=+43.825462543"
Jul 31 23:46:58.861302 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.861267    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-5dmjq" podStartSLOduration=29.477675426 podStartE2EDuration="36.861256414s" totalImagesPullingTime="7.383580988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.842176338 +0000 UTC m=+43.807371962" watchObservedRunningTime="2026-07-31 23:46:58.861256414 +0000 UTC m=+43.826452037"
Jul 31 23:46:58.891161 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.891101    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dr4vj" podStartSLOduration=1.505951002 podStartE2EDuration="8.891081893s" totalImagesPullingTime="7.385130891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.889893454 +0000 UTC m=+43.855089078" watchObservedRunningTime="2026-07-31 23:46:58.891081893 +0000 UTC m=+43.856277515"
Jul 31 23:46:58.891710 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.891676    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2kqrw" podStartSLOduration=29.652503405 podStartE2EDuration="36.891663309s" totalImagesPullingTime="7.239159904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.875618635 +0000 UTC m=+43.840814259" watchObservedRunningTime="2026-07-31 23:46:58.891663309 +0000 UTC m=+43.856858934"
Jul 31 23:46:58.907847 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.907783    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj" podStartSLOduration=29.410726438 podStartE2EDuration="36.907760935s" totalImagesPullingTime="7.497034497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.905838787 +0000 UTC m=+43.871034411" watchObservedRunningTime="2026-07-31 23:46:58.907760935 +0000 UTC m=+43.872956550"
Jul 31 23:46:58.955234 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.955105    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" podStartSLOduration=29.567412352 podStartE2EDuration="36.95508223s" totalImagesPullingTime="7.387669878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.934900835 +0000 UTC m=+43.900096460" watchObservedRunningTime="2026-07-31 23:46:58.95508223 +0000 UTC m=+43.920277855"
Jul 31 23:46:58.983573 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:58.983430    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.037346 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.037309    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.574754 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.574695    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-g2zqs" podStartSLOduration=30.193061024 podStartE2EDuration="37.574670837s" totalImagesPullingTime="7.381609813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:58.956394215 +0000 UTC m=+43.921589819" watchObservedRunningTime="2026-07-31 23:46:59.574670837 +0000 UTC m=+44.539866462"
Jul 31 23:46:59.575124 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.575012    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dxz7r"]
Jul 31 23:46:59.593326 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.593294    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.601425 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.601392    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxz7r"]
Jul 31 23:46:59.601601 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.601530    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.603734 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.603699    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:59.603894 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.603875    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fst58\""
Jul 31 23:46:59.603980 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.603911    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:59.632082 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.632044    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.632082 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.632092    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-crio-socket\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.632306 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.632167    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-data-volume\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.632341 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.632300    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5f2z\" (UniqueName: \"kubernetes.io/projected/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-api-access-t5f2z\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.632378 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.632364    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.656109 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.656021    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.733747 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.733706    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.733925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.733761    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-crio-socket\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.733925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.733814    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-data-volume\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.733925 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:59.733864    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:59.733925 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.733910    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-crio-socket\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.733925 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:59.733926    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:00.233910302 +0000 UTC m=+45.199105938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:59.734106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.733944    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5f2z\" (UniqueName: \"kubernetes.io/projected/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-api-access-t5f2z\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.734106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.734011    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.734181 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:59.734113    2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:59.734181 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:46:59.734146    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:00.234136999 +0000 UTC m=+45.199332614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:59.734260 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.734223    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-data-volume\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.744107 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.744075    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5f2z\" (UniqueName: \"kubernetes.io/projected/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-api-access-t5f2z\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:46:59.841537 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.841416    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.845363 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.845342    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/0.log"
Jul 31 23:46:59.845528 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.845379    2536 generic.go:356] "Generic (PLEG): container finished" podID="7e6b3eb5-7d66-4655-b38f-863883402a84" containerID="09bbd2d3df9c25dccdb9625503ba8ef82c8aad4356e369868a9aceeea3078109" exitCode=255
Jul 31 23:46:59.845528 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.845408    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" event={"ID":"7e6b3eb5-7d66-4655-b38f-863883402a84","Type":"ContainerDied","Data":"09bbd2d3df9c25dccdb9625503ba8ef82c8aad4356e369868a9aceeea3078109"}
Jul 31 23:46:59.846119 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.846076    2536 scope.go:117] "RemoveContainer" containerID="09bbd2d3df9c25dccdb9625503ba8ef82c8aad4356e369868a9aceeea3078109"
Jul 31 23:46:59.938129 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:46:59.938084    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.034782 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.034747    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.094185 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.094103    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.238950 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.238917    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:00.239134 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.238986    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:00.239134 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:00.239044    2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:47:00.239134 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:00.239105    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:00.239134 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:00.239115    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:01.239096998 +0000 UTC m=+46.204292600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : configmap "kube-rbac-proxy" not found
Jul 31 23:47:00.239278 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:00.239154    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:01.239137677 +0000 UTC m=+46.204333296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:00.541974 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.541930    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.641052 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.641019    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.743783 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.743745    2536 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:47:00.849496 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.849411    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:47:00.849829 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.849812    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/0.log"
Jul 31 23:47:00.849887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.849848    2536 generic.go:356] "Generic (PLEG): container finished" podID="7e6b3eb5-7d66-4655-b38f-863883402a84" containerID="13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88" exitCode=255
Jul 31 23:47:00.849941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.849888    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" event={"ID":"7e6b3eb5-7d66-4655-b38f-863883402a84","Type":"ContainerDied","Data":"13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88"}
Jul 31 23:47:00.849941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.849932    2536 scope.go:117] "RemoveContainer" containerID="09bbd2d3df9c25dccdb9625503ba8ef82c8aad4356e369868a9aceeea3078109"
Jul 31 23:47:00.850242 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:00.850222    2536 scope.go:117] "RemoveContainer" containerID="13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88"
Jul 31 23:47:00.850426 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:00.850404    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zdghk_openshift-console-operator(7e6b3eb5-7d66-4655-b38f-863883402a84)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" podUID="7e6b3eb5-7d66-4655-b38f-863883402a84"
Jul 31 23:47:01.248866 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.248784    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:01.248999 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:01.248943    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:01.248999 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:01.248990    2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:47:01.249071 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.248949    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:01.249071 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:01.249016    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.248994794 +0000 UTC m=+48.214190398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:01.249145 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:01.249082    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.249060502 +0000 UTC m=+48.214256106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : configmap "kube-rbac-proxy" not found
Jul 31 23:47:01.670885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.670852    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.771596 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.771557    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.843530 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.843502    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zlm46_dfcb504d-a872-4bfc-ad0c-976806a7def6/dns-node-resolver/0.log"
Jul 31 23:47:01.854572 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.854545    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:47:01.854887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:01.854863    2536 scope.go:117] "RemoveContainer" containerID="13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88"
Jul 31 23:47:01.855032 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:01.855013    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zdghk_openshift-console-operator(7e6b3eb5-7d66-4655-b38f-863883402a84)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" podUID="7e6b3eb5-7d66-4655-b38f-863883402a84"
Jul 31 23:47:02.244401 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:02.244359    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.346028 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:02.345994    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.446642 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:02.446615    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmzcw_40464426-09cc-4746-93bc-2ec417a8b4c2/node-ca/0.log"
Jul 31 23:47:02.448034 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:02.448005    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.620986 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:02.620947    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.011230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:03.011137    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.267521 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:03.267410    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:03.267701 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:03.267555    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:03.267701 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:03.267599    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:03.267701 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:03.267628    2536 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:47:03.267701 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:03.267701    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:07.267678023 +0000 UTC m=+52.232873641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:03.267861 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:03.267725    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:07.267715134 +0000 UTC m=+52.232910734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : configmap "kube-rbac-proxy" not found
Jul 31 23:47:03.444893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:03.444863    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr4vj_ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150/serve-healthcheck-canary/0.log"
Jul 31 23:47:05.742679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:05.742640    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.290496 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.290460    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.296555 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296521    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:47:06.296679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296590    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:47:06.296722 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296687    2536 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:47:06.296756 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296716    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:06.296787 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296759    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls podName:a39d93a8-1777-4329-ab31-24a680418b44 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.296739278 +0000 UTC m=+67.261934907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cs2kt" (UID: "a39d93a8-1777-4329-ab31-24a680418b44") : secret "samples-operator-tls" not found
Jul 31 23:47:06.296787 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296687    2536 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 31 23:47:06.296860 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296813    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:06.296860 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296827    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert podName:316ac0b7-e394-405e-b5eb-3c5575a64dbe nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.296809668 +0000 UTC m=+67.262005291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zgvjl" (UID: "316ac0b7-e394-405e-b5eb-3c5575a64dbe") : secret "networking-console-plugin-cert" not found
Jul 31 23:47:06.296860 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296855    2536 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296863    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.296852216 +0000 UTC m=+67.262047833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : configmap references non-existent config key: service-ca.crt
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296884    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296908    2536 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296916    2536 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76db5bc764-v6hj5: secret "image-registry-tls" not found
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.296920    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:47:06.296982 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296938    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls podName:2eac1e71-b840-4cc3-8c3f-0a45a8595420 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.296930171 +0000 UTC m=+67.262125774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls") pod "image-registry-76db5bc764-v6hj5" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420") : secret "image-registry-tls" not found
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.296984    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls podName:d563d196-7d23-4c7c-adc7-4f1c8b10f1c7 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.296951139 +0000 UTC m=+67.262146754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls") pod "dns-default-cgcbc" (UID: "d563d196-7d23-4c7c-adc7-4f1c8b10f1c7") : secret "dns-default-metrics-tls" not found
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.297001    2536 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.297027    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.297036    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls podName:75c48d66-57f1-495e-97d3-8f735cf965fe nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.297026169 +0000 UTC m=+67.262221795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-hpkqg" (UID: "75c48d66-57f1-495e-97d3-8f735cf965fe") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.297088    2536 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:47:06.297221 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:06.297139    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs podName:cbfa1dd6-1203-4c70-8e8c-48b5d95814a0 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.297128486 +0000 UTC m=+67.262324103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs") pod "router-default-6c4768f676-zrpn9" (UID: "cbfa1dd6-1203-4c70-8e8c-48b5d95814a0") : secret "router-metrics-certs-default" not found
Jul 31 23:47:06.394962 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.394922    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.493606 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.493567    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.594823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.594736    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.694034 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.693998    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.810261 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.810226    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.995608 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:06.995521    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.307046 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:07.307008    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:07.307233 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:07.307073    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:07.307284 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:07.307260    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:07.307342 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:07.307331    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:15.307312145 +0000 UTC m=+60.272507767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:07.307568 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:07.307549    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:08.194204 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.194159    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.398034 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.397991    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.597542 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.597505    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.793470 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.793432    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.838624 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.838591    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:47:08.839016 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.839001    2536 scope.go:117] "RemoveContainer" containerID="13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88"
Jul 31 23:47:08.839169 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:08.839150    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zdghk_openshift-console-operator(7e6b3eb5-7d66-4655-b38f-863883402a84)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" podUID="7e6b3eb5-7d66-4655-b38f-863883402a84"
Jul 31 23:47:08.992535 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:08.992421    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.093137 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:09.093093    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.993262 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:09.993228    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.094888 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.094850    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.388430 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.388393    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.443786 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.443749    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.591819 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.591781    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.743185 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.743088    2536 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:47:10.743476 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.743460    2536 scope.go:117] "RemoveContainer" containerID="13170c52c51190942d3c8fbba9b191d00e647f1415d5ced7eeca7307c3e1de88"
Jul 31 23:47:10.792777 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.792743    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.884664 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.884637    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:47:10.884845 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.884696    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk" event={"ID":"7e6b3eb5-7d66-4655-b38f-863883402a84","Type":"ContainerStarted","Data":"7577a5d74e86e9370ee11a2caf7e329e4f1a0efe63a042ec5ebfa5bdeb1a4cbc"}
Jul 31 23:47:10.884991 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.884960    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:47:10.993081 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:10.993048    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.046324 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.046286    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.425111 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.425031    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zdghk"
Jul 31 23:47:11.595859 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.595825    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.697320 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.697240    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.797148 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.797117    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.896897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:11.896863    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.023949 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.023860    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.126995 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.126956    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.227156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.227113    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.325012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.324983    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.424989 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.424956    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.525455 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.525420    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.625418 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.625334    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.726393 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.726362    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.827446 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:12.827415    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.089276 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:13.089238    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.393588 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:13.393481    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.685380 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:13.685291    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.792523 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:13.792467    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.993565 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:13.993461    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:14.113762 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:14.113730    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:14.215455 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:14.215419    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:14.390404 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:14.390368    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.195077 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:15.195040    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.380834 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:15.380797    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:15.381013 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:15.380913    2536 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:15.381013 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:15.380970    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls podName:2de10c36-1b46-406c-91e3-5ca95e0d0d7a nodeName:}" failed. No retries permitted until 2026-07-31 23:47:31.380956461 +0000 UTC m=+76.346152075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxz7r" (UID: "2de10c36-1b46-406c-91e3-5ca95e0d0d7a") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:15.396399 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:15.396361    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.604880 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:15.604841    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.794153 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:15.794107    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.728470 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:16.728445    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d9cnv"
Jul 31 23:47:16.795472 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:16.795439    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.991498 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:16.991388    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.195296 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:17.195255    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.395507 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:17.395450    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.509711 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:19.509675    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.577978 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:19.577943    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:21.061852 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.061813    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:21.115047 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.115002    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:21.333525 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.333414    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:47:21.333525 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.333503    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:47:21.333755 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.333535    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:47:21.336022 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.335992    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wptl\" (UniqueName: \"kubernetes.io/projected/0fc4f897-6876-4f7e-a1d5-f897546761fb-kube-api-access-5wptl\") pod \"network-check-target-8kmsg\" (UID: \"0fc4f897-6876-4f7e-a1d5-f897546761fb\") " pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:47:21.336147 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.336044    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:21.336147 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.336081    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:21.346691 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.346654    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5475fe4-6ba2-4652-84f5-0f5a887b4a0e-original-pull-secret\") pod \"global-pull-secret-syncer-dx87q\" (UID: \"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e\") " pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:47:21.346820 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.346718    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0eac260-1645-44ba-9ed6-549540cdef6d-metrics-certs\") pod \"network-metrics-daemon-zq5g7\" (UID: \"d0eac260-1645-44ba-9ed6-549540cdef6d\") " pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:47:21.464391 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.464352    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx87q"
Jul 31 23:47:21.574734 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.574707    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5t5tm\""
Jul 31 23:47:21.582164 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.582136    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v7qx8\""
Jul 31 23:47:21.583122 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.583097    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:47:21.590288 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.590267    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq5g7"
Jul 31 23:47:21.597108 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.597084    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dx87q"]
Jul 31 23:47:21.600082 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:21.600049    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5475fe4_6ba2_4652_84f5_0f5a887b4a0e.slice/crio-1f73dd5944ee6da13754930f3dec5ac86ce479cb84df9df69e2400cc9f0ad00b WatchSource:0}: Error finding container 1f73dd5944ee6da13754930f3dec5ac86ce479cb84df9df69e2400cc9f0ad00b: Status 404 returned error can't find the container with id 1f73dd5944ee6da13754930f3dec5ac86ce479cb84df9df69e2400cc9f0ad00b
Jul 31 23:47:21.722770 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.722733    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8kmsg"]
Jul 31 23:47:21.725673 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:21.725644    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc4f897_6876_4f7e_a1d5_f897546761fb.slice/crio-74184114baa35f60a9ab5b7749b750c794fced78f6439171b26268d320be0ea6 WatchSource:0}: Error finding container 74184114baa35f60a9ab5b7749b750c794fced78f6439171b26268d320be0ea6: Status 404 returned error can't find the container with id 74184114baa35f60a9ab5b7749b750c794fced78f6439171b26268d320be0ea6
Jul 31 23:47:21.741526 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.741477    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq5g7"]
Jul 31 23:47:21.744202 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:21.744168    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0eac260_1645_44ba_9ed6_549540cdef6d.slice/crio-8bd2d0c0fe5bf5d0edbe767cc630f1336141c826134bcd23a77a1f63bd71c1ae WatchSource:0}: Error finding container 8bd2d0c0fe5bf5d0edbe767cc630f1336141c826134bcd23a77a1f63bd71c1ae: Status 404 returned error can't find the container with id 8bd2d0c0fe5bf5d0edbe767cc630f1336141c826134bcd23a77a1f63bd71c1ae
Jul 31 23:47:21.925949 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.925843    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8kmsg" event={"ID":"0fc4f897-6876-4f7e-a1d5-f897546761fb","Type":"ContainerStarted","Data":"ac16e36b3d8ed8de1a97e7cba633c808606e96e901c368d843c4c1d9aa684b14"}
Jul 31 23:47:21.925949 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.925900    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8kmsg" event={"ID":"0fc4f897-6876-4f7e-a1d5-f897546761fb","Type":"ContainerStarted","Data":"74184114baa35f60a9ab5b7749b750c794fced78f6439171b26268d320be0ea6"}
Jul 31 23:47:21.925949 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.925921    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:47:21.926973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.926950    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq5g7" event={"ID":"d0eac260-1645-44ba-9ed6-549540cdef6d","Type":"ContainerStarted","Data":"8bd2d0c0fe5bf5d0edbe767cc630f1336141c826134bcd23a77a1f63bd71c1ae"}
Jul 31 23:47:21.928056 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.928034    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dx87q" event={"ID":"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e","Type":"ContainerStarted","Data":"1f73dd5944ee6da13754930f3dec5ac86ce479cb84df9df69e2400cc9f0ad00b"}
Jul 31 23:47:21.940035 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:21.939963    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8kmsg" podStartSLOduration=66.939947742 podStartE2EDuration="1m6.939947742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:21.939426663 +0000 UTC m=+66.904622287" watchObservedRunningTime="2026-07-31 23:47:21.939947742 +0000 UTC m=+66.905143343"
Jul 31 23:47:22.344418 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344377    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344454    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344541    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344583    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344611    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344640    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:47:22.344897 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.344668    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:22.345914 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.345850    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-service-ca-bundle\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:22.348416 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.348352    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/316ac0b7-e394-405e-b5eb-3c5575a64dbe-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zgvjl\" (UID: \"316ac0b7-e394-405e-b5eb-3c5575a64dbe\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:47:22.349263 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.349212    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c48d66-57f1-495e-97d3-8f735cf965fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-hpkqg\" (UID: \"75c48d66-57f1-495e-97d3-8f735cf965fe\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:47:22.349964 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.349911    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"image-registry-76db5bc764-v6hj5\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") " pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:22.351638 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.351614    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfa1dd6-1203-4c70-8e8c-48b5d95814a0-metrics-certs\") pod \"router-default-6c4768f676-zrpn9\" (UID: \"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0\") " pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:22.351748 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.351618    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a39d93a8-1777-4329-ab31-24a680418b44-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cs2kt\" (UID: \"a39d93a8-1777-4329-ab31-24a680418b44\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:47:22.351936 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.351915    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d563d196-7d23-4c7c-adc7-4f1c8b10f1c7-metrics-tls\") pod \"dns-default-cgcbc\" (UID: \"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7\") " pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:22.460668 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.460463    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.512626 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.512584    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxfp4\""
Jul 31 23:47:22.521081 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.521045    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:22.521514 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.521472    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rbrbb\""
Jul 31 23:47:22.530232 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.530198    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"
Jul 31 23:47:22.550802 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.550763    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.560347 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.560314    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br2vn\""
Jul 31 23:47:22.569142 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.569112    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"
Jul 31 23:47:22.580169 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.580145    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rs996\""
Jul 31 23:47:22.589049 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.589017    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:22.593619 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.593590    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dkrqj\""
Jul 31 23:47:22.598982 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.598917    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w7zwr\""
Jul 31 23:47:22.601689 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.601659    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:22.607931 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:22.607903    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"
Jul 31 23:47:23.087425 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.087336    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:47:23.106223 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.106141    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg"]
Jul 31 23:47:23.118246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.118144    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgcbc"]
Jul 31 23:47:23.206936 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:23.206842    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eac1e71_b840_4cc3_8c3f_0a45a8595420.slice/crio-908c63b7046b6f78c71ce54c2d89e828b62233c41157686dff7b17f0c6c5650f WatchSource:0}: Error finding container 908c63b7046b6f78c71ce54c2d89e828b62233c41157686dff7b17f0c6c5650f: Status 404 returned error can't find the container with id 908c63b7046b6f78c71ce54c2d89e828b62233c41157686dff7b17f0c6c5650f
Jul 31 23:47:23.207399 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:23.207362    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c48d66_57f1_495e_97d3_8f735cf965fe.slice/crio-e8c0c3973106aba5e5c38d87c789b18a4c9a7a3f86df35a4d4627ffb01d488dc WatchSource:0}: Error finding container e8c0c3973106aba5e5c38d87c789b18a4c9a7a3f86df35a4d4627ffb01d488dc: Status 404 returned error can't find the container with id e8c0c3973106aba5e5c38d87c789b18a4c9a7a3f86df35a4d4627ffb01d488dc
Jul 31 23:47:23.208555 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:23.208530    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd563d196_7d23_4c7c_adc7_4f1c8b10f1c7.slice/crio-7b7d77e6fb1376ed21ae6b5ccfdb0a6c28dbeef0ec8d3d0e72e947a3ded7c52c WatchSource:0}: Error finding container 7b7d77e6fb1376ed21ae6b5ccfdb0a6c28dbeef0ec8d3d0e72e947a3ded7c52c: Status 404 returned error can't find the container with id 7b7d77e6fb1376ed21ae6b5ccfdb0a6c28dbeef0ec8d3d0e72e947a3ded7c52c
Jul 31 23:47:23.353474 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.353372    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl"]
Jul 31 23:47:23.358156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.358129    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt"]
Jul 31 23:47:23.359364 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.359296    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c4768f676-zrpn9"]
Jul 31 23:47:23.365768 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:23.365736    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316ac0b7_e394_405e_b5eb_3c5575a64dbe.slice/crio-7ac807e9e0f803aecb48ad72a64b89624ca48e6d8184d6d0732e22521c5bf3ac WatchSource:0}: Error finding container 7ac807e9e0f803aecb48ad72a64b89624ca48e6d8184d6d0732e22521c5bf3ac: Status 404 returned error can't find the container with id 7ac807e9e0f803aecb48ad72a64b89624ca48e6d8184d6d0732e22521c5bf3ac
Jul 31 23:47:23.366397 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:23.366369    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfa1dd6_1203_4c70_8e8c_48b5d95814a0.slice/crio-ab8fa8920d1c1b0f226b179feab019515e783da36fede27e3b65a2acf9bdce9b WatchSource:0}: Error finding container ab8fa8920d1c1b0f226b179feab019515e783da36fede27e3b65a2acf9bdce9b: Status 404 returned error can't find the container with id ab8fa8920d1c1b0f226b179feab019515e783da36fede27e3b65a2acf9bdce9b
Jul 31 23:47:23.942858 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.942751    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl" event={"ID":"316ac0b7-e394-405e-b5eb-3c5575a64dbe","Type":"ContainerStarted","Data":"7ac807e9e0f803aecb48ad72a64b89624ca48e6d8184d6d0732e22521c5bf3ac"}
Jul 31 23:47:23.949973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.949154    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c4768f676-zrpn9" event={"ID":"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0","Type":"ContainerStarted","Data":"e522fcc25090087c03425e977be46794784dac52319f6ed50dc5b6e488a14235"}
Jul 31 23:47:23.949973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.949214    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c4768f676-zrpn9" event={"ID":"cbfa1dd6-1203-4c70-8e8c-48b5d95814a0","Type":"ContainerStarted","Data":"ab8fa8920d1c1b0f226b179feab019515e783da36fede27e3b65a2acf9bdce9b"}
Jul 31 23:47:23.955653 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.955618    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq5g7" event={"ID":"d0eac260-1645-44ba-9ed6-549540cdef6d","Type":"ContainerStarted","Data":"042c1840d95ff8b740dddf77e1e5adce20a7abb69e03c6caaabc35496ce049b4"}
Jul 31 23:47:23.955821 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.955684    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq5g7" event={"ID":"d0eac260-1645-44ba-9ed6-549540cdef6d","Type":"ContainerStarted","Data":"2a609306607ba1a7b7963f29929aa14e931873f4009b6a8f12a2d3c4b194da29"}
Jul 31 23:47:23.959729 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.959618    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgcbc" event={"ID":"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7","Type":"ContainerStarted","Data":"7b7d77e6fb1376ed21ae6b5ccfdb0a6c28dbeef0ec8d3d0e72e947a3ded7c52c"}
Jul 31 23:47:23.962782 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.962619    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt" event={"ID":"a39d93a8-1777-4329-ab31-24a680418b44","Type":"ContainerStarted","Data":"35a36b0f48f43651cb46319e46d03b97df0c6b367ba7e18f3774306061950d58"}
Jul 31 23:47:23.969513 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.968594    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c4768f676-zrpn9" podStartSLOduration=61.968574125 podStartE2EDuration="1m1.968574125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:23.967991495 +0000 UTC m=+68.933187116" watchObservedRunningTime="2026-07-31 23:47:23.968574125 +0000 UTC m=+68.933769748"
Jul 31 23:47:23.972788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.972078    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" event={"ID":"2eac1e71-b840-4cc3-8c3f-0a45a8595420","Type":"ContainerStarted","Data":"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"}
Jul 31 23:47:23.972788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.972131    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" event={"ID":"2eac1e71-b840-4cc3-8c3f-0a45a8595420","Type":"ContainerStarted","Data":"908c63b7046b6f78c71ce54c2d89e828b62233c41157686dff7b17f0c6c5650f"}
Jul 31 23:47:23.972788 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.972732    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:23.974909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.974847    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg" event={"ID":"75c48d66-57f1-495e-97d3-8f735cf965fe","Type":"ContainerStarted","Data":"e8c0c3973106aba5e5c38d87c789b18a4c9a7a3f86df35a4d4627ffb01d488dc"}
Jul 31 23:47:23.985284 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:23.984448    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zq5g7" podStartSLOduration=67.488250936 podStartE2EDuration="1m8.984426138s" totalImagesPullingTime="1.496175202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:23.983685354 +0000 UTC m=+68.948880978" watchObservedRunningTime="2026-07-31 23:47:23.984426138 +0000 UTC m=+68.949621763"
Jul 31 23:47:24.003970 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.002454    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" podStartSLOduration=68.002430663 podStartE2EDuration="1m8.002430663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:24.002250871 +0000 UTC m=+68.967446494" watchObservedRunningTime="2026-07-31 23:47:24.002430663 +0000 UTC m=+68.967626288"
Jul 31 23:47:24.073683 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.073499    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.173270 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.173201    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.602194 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.602158    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:24.605381 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.605313    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:24.979746 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.979653    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:24.981308 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:24.981282    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c4768f676-zrpn9"
Jul 31 23:47:25.689909 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:25.689870    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.791902 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:25.791851    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:26.907848 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:26.907782    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.510716 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:28.510659    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.561471 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:28.561414    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.998276 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:28.998240    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl" event={"ID":"316ac0b7-e394-405e-b5eb-3c5575a64dbe","Type":"ContainerStarted","Data":"309b1f47cf63c63738fb15e77900a4753ce39a724220e19750952a7b0d6c195e"}
Jul 31 23:47:29.000452 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.000385    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dx87q" event={"ID":"f5475fe4-6ba2-4652-84f5-0f5a887b4a0e","Type":"ContainerStarted","Data":"be1d5d1f8a27b270f04930cb05157066bfa16ca1d627264077ca9dec4f29438b"}
Jul 31 23:47:29.002401 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.002318    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgcbc" event={"ID":"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7","Type":"ContainerStarted","Data":"4e27da25af6fe0fb9a8cfe464c72bb27066142a75b5a45f04497c4cd83fdb454"}
Jul 31 23:47:29.004346 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.004319    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt" event={"ID":"a39d93a8-1777-4329-ab31-24a680418b44","Type":"ContainerStarted","Data":"2abba4d8e2e19d52409787f375a2d71e66fbe37a2e8fc352514f787fb7602d56"}
Jul 31 23:47:29.007198 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.007067    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg" event={"ID":"75c48d66-57f1-495e-97d3-8f735cf965fe","Type":"ContainerStarted","Data":"0082e0c023dd9b8f8c440bb22c42f87895edd89614718baa09c11e3523a25a4b"}
Jul 31 23:47:29.045345 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.043746    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dx87q" podStartSLOduration=65.860443531 podStartE2EDuration="1m13.043723933s" totalImagesPullingTime="7.183280402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:29.026077012 +0000 UTC m=+73.991272637" watchObservedRunningTime="2026-07-31 23:47:29.043723933 +0000 UTC m=+74.008919558"
Jul 31 23:47:29.045345 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.044985    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-hpkqg" podStartSLOduration=61.497664881 podStartE2EDuration="1m7.044971467s" totalImagesPullingTime="5.547306586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:29.043105264 +0000 UTC m=+74.008300888" watchObservedRunningTime="2026-07-31 23:47:29.044971467 +0000 UTC m=+74.010167090"
Jul 31 23:47:29.065634 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.065570    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zgvjl" podStartSLOduration=39.648631263 podStartE2EDuration="45.065542488s" totalImagesPullingTime="5.416911225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:29.063425177 +0000 UTC m=+74.028620801" watchObservedRunningTime="2026-07-31 23:47:29.065542488 +0000 UTC m=+74.030738090"
Jul 31 23:47:29.177336 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.177301    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.275006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.274967    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.401384 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.401354    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.404726 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.404700    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"]
Jul 31 23:47:29.429310 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.429263    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"]
Jul 31 23:47:29.429527 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.429375    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:29.431681 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.431654    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 31 23:47:29.431819 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.431801    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k67gv\""
Jul 31 23:47:29.509856 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.509822    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-tmmsj\" (UID: \"67fb135f-4240-445a-a77b-49fc67eca297\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:29.611247 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:29.611203    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-tmmsj\" (UID: \"67fb135f-4240-445a-a77b-49fc67eca297\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:29.611644 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:29.611350    2536 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:47:29.611644 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:29.611417    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates podName:67fb135f-4240-445a-a77b-49fc67eca297 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:30.111401572 +0000 UTC m=+75.076597173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-tmmsj" (UID: "67fb135f-4240-445a-a77b-49fc67eca297") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:47:30.013533 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.013495    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgcbc" event={"ID":"d563d196-7d23-4c7c-adc7-4f1c8b10f1c7","Type":"ContainerStarted","Data":"f15563aaaba7666ae3fcf66b228c3a289dd46bbe9059af6e554e173a9e86ec29"}
Jul 31 23:47:30.013753 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.013695    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:30.015501 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.015459    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt" event={"ID":"a39d93a8-1777-4329-ab31-24a680418b44","Type":"ContainerStarted","Data":"4cddf73aa34e1a2c1e8f472801019dfc0afa84de0197faa75331df5136ef4fd2"}
Jul 31 23:47:30.032832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.032779    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cgcbc" podStartSLOduration=34.486154936 podStartE2EDuration="40.032761703s" totalImagesPullingTime="5.546606767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:30.031718344 +0000 UTC m=+74.996913961" watchObservedRunningTime="2026-07-31 23:47:30.032761703 +0000 UTC m=+74.997957329"
Jul 31 23:47:30.047683 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.047628    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cs2kt" podStartSLOduration=62.685059555 podStartE2EDuration="1m8.047612268s" totalImagesPullingTime="5.362552713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:30.046645194 +0000 UTC m=+75.011840816" watchObservedRunningTime="2026-07-31 23:47:30.047612268 +0000 UTC m=+75.012807890"
Jul 31 23:47:30.115021 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.114979    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-tmmsj\" (UID: \"67fb135f-4240-445a-a77b-49fc67eca297\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:30.115231 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:30.115140    2536 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:47:30.115231 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:30.115224    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates podName:67fb135f-4240-445a-a77b-49fc67eca297 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:31.115201325 +0000 UTC m=+76.080396941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-tmmsj" (UID: "67fb135f-4240-445a-a77b-49fc67eca297") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:47:30.306443 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.306404    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:30.409649 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:30.409614    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:31.123006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.122962    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-tmmsj\" (UID: \"67fb135f-4240-445a-a77b-49fc67eca297\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:31.125380 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.125351    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/67fb135f-4240-445a-a77b-49fc67eca297-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-tmmsj\" (UID: \"67fb135f-4240-445a-a77b-49fc67eca297\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:31.238603 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.238551    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:31.366094 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.366061    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"]
Jul 31 23:47:31.369038 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:31.369001    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb135f_4240_445a_a77b_49fc67eca297.slice/crio-82d5030ec1a1b4c83ebcc5a4c8e1922b7a337b9d76a99c07ac75d10e23b7fee9 WatchSource:0}: Error finding container 82d5030ec1a1b4c83ebcc5a4c8e1922b7a337b9d76a99c07ac75d10e23b7fee9: Status 404 returned error can't find the container with id 82d5030ec1a1b4c83ebcc5a4c8e1922b7a337b9d76a99c07ac75d10e23b7fee9
Jul 31 23:47:31.424731 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.424634    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:31.426987 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.426962    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2de10c36-1b46-406c-91e3-5ca95e0d0d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxz7r\" (UID: \"2de10c36-1b46-406c-91e3-5ca95e0d0d7a\") " pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:31.715776 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.715691    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fst58\""
Jul 31 23:47:31.724471 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.724436    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxz7r"
Jul 31 23:47:31.860637 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.860566    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxz7r"]
Jul 31 23:47:31.865357 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:31.865317    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de10c36_1b46_406c_91e3_5ca95e0d0d7a.slice/crio-976f651dfb3093e700b1db8afb390062dc5727b9c0cef5c03af978d104921e3c WatchSource:0}: Error finding container 976f651dfb3093e700b1db8afb390062dc5727b9c0cef5c03af978d104921e3c: Status 404 returned error can't find the container with id 976f651dfb3093e700b1db8afb390062dc5727b9c0cef5c03af978d104921e3c
Jul 31 23:47:31.913213 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:31.913178    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.013796 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.013560    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.024879 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.024838    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj" event={"ID":"67fb135f-4240-445a-a77b-49fc67eca297","Type":"ContainerStarted","Data":"82d5030ec1a1b4c83ebcc5a4c8e1922b7a337b9d76a99c07ac75d10e23b7fee9"}
Jul 31 23:47:32.026605 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.026576    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxz7r" event={"ID":"2de10c36-1b46-406c-91e3-5ca95e0d0d7a","Type":"ContainerStarted","Data":"d073bc6bbb2eb29e62de3812539ac7ed550237920373be0e60b8df5bf030ff2d"}
Jul 31 23:47:32.026731 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.026638    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxz7r" event={"ID":"2de10c36-1b46-406c-91e3-5ca95e0d0d7a","Type":"ContainerStarted","Data":"976f651dfb3093e700b1db8afb390062dc5727b9c0cef5c03af978d104921e3c"}
Jul 31 23:47:32.319375 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.319337    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.445237 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.445200    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.831163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.831121    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.931667 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:32.931587    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:33.032753 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.032646    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj" event={"ID":"67fb135f-4240-445a-a77b-49fc67eca297","Type":"ContainerStarted","Data":"9ac3f2d99e56c5b14abb622b4935e75371a7c9f97eeaced1f4304ac27e368bb8"}
Jul 31 23:47:33.032937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.032839    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:33.035693 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.035647    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:33.035878 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.035856    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxz7r" event={"ID":"2de10c36-1b46-406c-91e3-5ca95e0d0d7a","Type":"ContainerStarted","Data":"f9ccbaa9a3c2ad21f72a889604edad4cfcf9eb9e82c816f1dd5a207dc748aa21"}
Jul 31 23:47:33.039240 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.039215    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj"
Jul 31 23:47:33.051677 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.051614    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-tmmsj" podStartSLOduration=3.063221175 podStartE2EDuration="4.051596475s" totalImagesPullingTime="988.3753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:33.050270853 +0000 UTC m=+78.015466476" watchObservedRunningTime="2026-07-31 23:47:33.051596475 +0000 UTC m=+78.016792097"
Jul 31 23:47:33.460161 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.460126    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:33.504780 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.504743    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-swkrv"]
Jul 31 23:47:33.507708 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.507677    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.511039 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.511009    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7p4kv\""
Jul 31 23:47:33.511227 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.511066    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 31 23:47:33.511227 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.511111    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:33.511227 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.511111    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 31 23:47:33.518035 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.518008    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-swkrv"]
Jul 31 23:47:33.525376 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.525330    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:33.644132 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.644090    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.644317 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.644158    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.644317 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.644187    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.644317 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.644219    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8rp\" (UniqueName: \"kubernetes.io/projected/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-kube-api-access-dn8rp\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.745263 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.745236    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.745396 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.745273    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.745396 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.745294    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.745396 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.745323    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8rp\" (UniqueName: \"kubernetes.io/projected/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-kube-api-access-dn8rp\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.745594 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:33.745406    2536 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 31 23:47:33.745594 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:33.745473    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls podName:5d4ef30d-1ae1-4a04-a26d-74e6c1267baf nodeName:}" failed. No retries permitted until 2026-07-31 23:47:34.245452924 +0000 UTC m=+79.210648539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-swkrv" (UID: "5d4ef30d-1ae1-4a04-a26d-74e6c1267baf") : secret "prometheus-operator-tls" not found
Jul 31 23:47:33.746022 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.746003    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.747856 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.747828    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.753470 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.753440    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8rp\" (UniqueName: \"kubernetes.io/projected/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-kube-api-access-dn8rp\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:33.998803 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:33.998704    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.041784 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.041738    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxz7r" event={"ID":"2de10c36-1b46-406c-91e3-5ca95e0d0d7a","Type":"ContainerStarted","Data":"220858f7e6fda72bde54105bad85d1d8b67a488cd976c05ec59f307721a6e559"}
Jul 31 23:47:34.061304 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.061246    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dxz7r" podStartSLOduration=33.317283026 podStartE2EDuration="35.061228998s" totalImagesPullingTime="1.743945972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:34.05956158 +0000 UTC m=+79.024757204" watchObservedRunningTime="2026-07-31 23:47:34.061228998 +0000 UTC m=+79.026424650"
Jul 31 23:47:34.250097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.249996    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:34.252465 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.252441    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d4ef30d-1ae1-4a04-a26d-74e6c1267baf-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-swkrv\" (UID: \"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:34.419632 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.419594    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv"
Jul 31 23:47:34.546990 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.546910    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-swkrv"]
Jul 31 23:47:34.549153 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:34.549126    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4ef30d_1ae1_4a04_a26d_74e6c1267baf.slice/crio-53f0adbec8edfb3d158e59ddc0fc430b956f64457aa4b4ee39260c5ecdebecc2 WatchSource:0}: Error finding container 53f0adbec8edfb3d158e59ddc0fc430b956f64457aa4b4ee39260c5ecdebecc2: Status 404 returned error can't find the container with id 53f0adbec8edfb3d158e59ddc0fc430b956f64457aa4b4ee39260c5ecdebecc2
Jul 31 23:47:34.707722 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.707690    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.791410 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:34.791374    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.047085 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:35.047000    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv" event={"ID":"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf","Type":"ContainerStarted","Data":"53f0adbec8edfb3d158e59ddc0fc430b956f64457aa4b4ee39260c5ecdebecc2"}
Jul 31 23:47:35.944107 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:35.944005    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:36.052559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:36.052519    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv" event={"ID":"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf","Type":"ContainerStarted","Data":"948c94b4aca3bf86aead572665b22d6a47dcff5a529eb435cb4f1af48e566557"}
Jul 31 23:47:36.052733 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:36.052585    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv" event={"ID":"5d4ef30d-1ae1-4a04-a26d-74e6c1267baf","Type":"ContainerStarted","Data":"dc9daaef4c8bb9d41a7241bd57265afba2a61f23b0b62a39151a6654d68a8f4d"}
Jul 31 23:47:36.078851 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:36.078794    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-swkrv" podStartSLOduration=1.9524732280000001 podStartE2EDuration="3.078776531s" totalImagesPullingTime="1.126303303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:36.077176954 +0000 UTC m=+81.042372579" watchObservedRunningTime="2026-07-31 23:47:36.078776531 +0000 UTC m=+81.043972152"
Jul 31 23:47:36.092552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:36.092518    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:36.192279 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:36.192242    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.111738 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.111695    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.626201 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.626145    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.866327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.866172    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.905356 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.905259    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t2hwq"]
Jul 31 23:47:37.908959 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.908933    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:37.911266 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.911243    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:37.912188 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.912172    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:37.912333 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.912314    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:37.912418 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.912311    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cmkpk\""
Jul 31 23:47:37.965860 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:37.965829    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.082885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.082842    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.082885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.082885    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2cvx\" (UniqueName: \"kubernetes.io/projected/c0b25a94-caff-47c6-ab2b-063f50800445-kube-api-access-s2cvx\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.082993    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083024    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-metrics-client-ca\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083043    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083106 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083057    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-root\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083242 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083136    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-textfile\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083242 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083169    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-wtmp\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.083242 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.083189    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-sys\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.126573 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.126538    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.183657 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183577    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183657 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183615    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-metrics-client-ca\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183657 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183632    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183721    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-root\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183785    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-textfile\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183806    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-root\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183819    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-wtmp\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.183881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183844    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-sys\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183890    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183938    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-sys\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183972    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-wtmp\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.183983    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2cvx\" (UniqueName: \"kubernetes.io/projected/c0b25a94-caff-47c6-ab2b-063f50800445-kube-api-access-s2cvx\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:38.183978    2536 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:38.184063    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls podName:c0b25a94-caff-47c6-ab2b-063f50800445 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:38.68404031 +0000 UTC m=+83.649235931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls") pod "node-exporter-t2hwq" (UID: "c0b25a94-caff-47c6-ab2b-063f50800445") : secret "node-exporter-tls" not found
Jul 31 23:47:38.184121 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.184098    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-textfile\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.184304    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-metrics-client-ca\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.184438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.184424    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.186147 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.186124    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.192309 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.192285    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2cvx\" (UniqueName: \"kubernetes.io/projected/c0b25a94-caff-47c6-ab2b-063f50800445-kube-api-access-s2cvx\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.228185 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.228153    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.328245 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.328210    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.430076 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.430038    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.529811 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.529724    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.689421 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.689385    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.691821 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.691798    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0b25a94-caff-47c6-ab2b-063f50800445-node-exporter-tls\") pod \"node-exporter-t2hwq\" (UID: \"c0b25a94-caff-47c6-ab2b-063f50800445\") " pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.718063 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.718027    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.819359 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.819323    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2hwq"
Jul 31 23:47:38.821536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.821499    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.834372 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:38.834329    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b25a94_caff_47c6_ab2b_063f50800445.slice/crio-ddd5c827b9efa454900993707ad9bd9b88ef340da594cd90b8af599c3c0c51ae WatchSource:0}: Error finding container ddd5c827b9efa454900993707ad9bd9b88ef340da594cd90b8af599c3c0c51ae: Status 404 returned error can't find the container with id ddd5c827b9efa454900993707ad9bd9b88ef340da594cd90b8af599c3c0c51ae
Jul 31 23:47:38.935936 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.935893    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.995751 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.995712    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:38.999847 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:38.999817    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.002313 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002286    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lh6bz\""
Jul 31 23:47:39.002474 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002327    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:47:39.002601 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002586    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:47:39.002654 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002591    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:47:39.002865 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002842    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:47:39.002942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002848    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:47:39.002942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002851    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:47:39.002942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002877    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:47:39.003068 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.002963    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:47:39.003156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.003138    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:47:39.017427 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.017399    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:39.020184 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.020154    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:39.064609 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.064575    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2hwq" event={"ID":"c0b25a94-caff-47c6-ab2b-063f50800445","Type":"ContainerStarted","Data":"ddd5c827b9efa454900993707ad9bd9b88ef340da594cd90b8af599c3c0c51ae"}
Jul 31 23:47:39.193736 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193644    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-web-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.193736 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193687    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193755    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-out\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193776    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193803    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193829    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193859    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193947    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.193973    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.194051    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.194103    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.194136    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.194218 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.194165    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9k46\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-kube-api-access-x9k46\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.294885 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.294843    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.294895    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.294936    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.294964    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.294999    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295024    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9k46\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-kube-api-access-x9k46\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295075 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295050    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-web-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295079    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:39.295099    2536 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295144    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-out\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:39.295179    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls podName:9e05d46a-9ef3-4434-89d3-63db02d44a79 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:39.795157291 +0000 UTC m=+84.760352903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9e05d46a-9ef3-4434-89d3-63db02d44a79") : secret "alertmanager-main-tls" not found
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295218    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295249    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295285    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295328    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.295876 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.295616    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.296513 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.296173    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.296944 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.296625    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e05d46a-9ef3-4434-89d3-63db02d44a79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.299335 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.299038    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.299335 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.299110    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-web-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.300038 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.299993    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-out\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.300535 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.300214    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.300703 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.300659    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.300987 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.300960    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.301583 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.301563    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.302104 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.302072    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.307286 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.307263    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9k46\" (UniqueName: \"kubernetes.io/projected/9e05d46a-9ef3-4434-89d3-63db02d44a79-kube-api-access-x9k46\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.799041 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.799001    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:39.799234 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:39.799147    2536 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 31 23:47:39.799234 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:47:39.799227    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls podName:9e05d46a-9ef3-4434-89d3-63db02d44a79 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:40.799207729 +0000 UTC m=+85.764403346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9e05d46a-9ef3-4434-89d3-63db02d44a79") : secret "alertmanager-main-tls" not found
Jul 31 23:47:39.894625 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:39.894590    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.021699 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.021610    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cgcbc"
Jul 31 23:47:40.071437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.071399    2536 generic.go:356] "Generic (PLEG): container finished" podID="c0b25a94-caff-47c6-ab2b-063f50800445" containerID="0261c25b19ccd18eaf345a061211501d6b572f7a742309243e53480580ade540" exitCode=0
Jul 31 23:47:40.071664 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.071450    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2hwq" event={"ID":"c0b25a94-caff-47c6-ab2b-063f50800445","Type":"ContainerDied","Data":"0261c25b19ccd18eaf345a061211501d6b572f7a742309243e53480580ade540"}
Jul 31 23:47:40.786020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.785977    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.808096 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.808058    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:40.810580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.810549    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e05d46a-9ef3-4434-89d3-63db02d44a79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e05d46a-9ef3-4434-89d3-63db02d44a79\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:40.841833 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.841787    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.940938 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.940902    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.941826 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.941801    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8fff5d969-5p94j"]
Jul 31 23:47:40.945754 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.945737    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:40.948273 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948252    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 23:47:40.948385 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948301    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f2lls799vo3ds\""
Jul 31 23:47:40.948385 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948321    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 23:47:40.948536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948446    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-96c7l\""
Jul 31 23:47:40.948605 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948587    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 23:47:40.948729 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948712    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 23:47:40.948801 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.948774    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 23:47:40.958711 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:40.958686    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8fff5d969-5p94j"]
Jul 31 23:47:41.014972 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.014938    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:41.078319 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.078230    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2hwq" event={"ID":"c0b25a94-caff-47c6-ab2b-063f50800445","Type":"ContainerStarted","Data":"cea227586bf4170a53c6b6d50807b86c270ead1dcb77bb1a1881efbdce9fd15c"}
Jul 31 23:47:41.078319 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.078294    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2hwq" event={"ID":"c0b25a94-caff-47c6-ab2b-063f50800445","Type":"ContainerStarted","Data":"5f4864352ff118fbe8f964b3db25017c4653a653d3a88ff237441434139a8023"}
Jul 31 23:47:41.098604 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.098548    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t2hwq" podStartSLOduration=3.206502238 podStartE2EDuration="4.098529885s" totalImagesPullingTime="823.832816ms" totalInitContainerRuntime="68.194831ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:41.097198742 +0000 UTC m=+86.062394367" watchObservedRunningTime="2026-07-31 23:47:41.098529885 +0000 UTC m=+86.063725553"
Jul 31 23:47:41.111192 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111165    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:41.111327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111274    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111312    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxv5\" (UniqueName: \"kubernetes.io/projected/fd92ec36-2398-44dc-b554-2a461bfb083d-kube-api-access-cwxv5\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111442 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111334    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111442 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111422    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111540 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111442    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-grpc-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111540 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111465    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd92ec36-2398-44dc-b554-2a461bfb083d-metrics-client-ca\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111540 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111523    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.111670 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.111570    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212018 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.211983    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212018 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212029    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxv5\" (UniqueName: \"kubernetes.io/projected/fd92ec36-2398-44dc-b554-2a461bfb083d-kube-api-access-cwxv5\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212195 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212062    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212195 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212158    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212195 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212183    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-grpc-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212341 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212215    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd92ec36-2398-44dc-b554-2a461bfb083d-metrics-client-ca\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212341 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212252    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.212341 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.212277    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.214872 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.214844    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd92ec36-2398-44dc-b554-2a461bfb083d-metrics-client-ca\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.216276 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.216208    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.218339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.217749    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.218339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.217895    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.218339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.217995    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.218339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.218063    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-grpc-tls\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.218339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.218262    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd92ec36-2398-44dc-b554-2a461bfb083d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.220830 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.220804    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxv5\" (UniqueName: \"kubernetes.io/projected/fd92ec36-2398-44dc-b554-2a461bfb083d-kube-api-access-cwxv5\") pod \"thanos-querier-8fff5d969-5p94j\" (UID: \"fd92ec36-2398-44dc-b554-2a461bfb083d\") " pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.256230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.256194    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:41.256464 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.256431    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:41.258639 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:41.258616    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e05d46a_9ef3_4434_89d3_63db02d44a79.slice/crio-37f24cfa3ea922e655a82ee60662b51d1ab6e9b45491d6e23177d83824ee92a2 WatchSource:0}: Error finding container 37f24cfa3ea922e655a82ee60662b51d1ab6e9b45491d6e23177d83824ee92a2: Status 404 returned error can't find the container with id 37f24cfa3ea922e655a82ee60662b51d1ab6e9b45491d6e23177d83824ee92a2
Jul 31 23:47:41.397801 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.397767    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:41.398365 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.398335    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8fff5d969-5p94j"]
Jul 31 23:47:41.400577 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:41.400549    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd92ec36_2398_44dc_b554_2a461bfb083d.slice/crio-e21143c6456278ca305a6efd763ae42aadf3ea8dba536b6a847985bd86fc3a07 WatchSource:0}: Error finding container e21143c6456278ca305a6efd763ae42aadf3ea8dba536b6a847985bd86fc3a07: Status 404 returned error can't find the container with id e21143c6456278ca305a6efd763ae42aadf3ea8dba536b6a847985bd86fc3a07
Jul 31 23:47:41.495568 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:41.495527    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:42.085082 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.085041    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"37f24cfa3ea922e655a82ee60662b51d1ab6e9b45491d6e23177d83824ee92a2"}
Jul 31 23:47:42.086693 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.086656    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"e21143c6456278ca305a6efd763ae42aadf3ea8dba536b6a847985bd86fc3a07"}
Jul 31 23:47:42.176402 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.176365    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cb6bf7c85-94hw6"]
Jul 31 23:47:42.180299 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.180223    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.182887 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.182816    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 23:47:42.183048 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.183028    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 23:47:42.184090 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.184065    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 23:47:42.184090 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.184079    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 23:47:42.184232 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.184080    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c1lgsu1c63r8o\""
Jul 31 23:47:42.184232 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.184104    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9wcg9\""
Jul 31 23:47:42.189746 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.189704    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cb6bf7c85-94hw6"]
Jul 31 23:47:42.221040 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221000    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221050    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bp5\" (UniqueName: \"kubernetes.io/projected/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-kube-api-access-58bp5\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221140    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-metrics-server-audit-profiles\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221230 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221203    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-client-certs\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221388 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221231    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-client-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221388 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221298    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-tls\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.221388 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.221318    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-audit-log\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322261 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322230    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-metrics-server-audit-profiles\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322299    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-client-certs\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322331    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-client-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322380    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-tls\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322410    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-audit-log\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322632 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322468    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.322632 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.322517    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bp5\" (UniqueName: \"kubernetes.io/projected/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-kube-api-access-58bp5\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.323066 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.323018    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-audit-log\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.323732 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.323330    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-metrics-server-audit-profiles\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.323732 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.323556    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.325700 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.325671    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-client-certs\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.325802 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.325773    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-secret-metrics-server-tls\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.326128 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.326106    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-client-ca-bundle\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.331966 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.331938    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bp5\" (UniqueName: \"kubernetes.io/projected/5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218-kube-api-access-58bp5\") pod \"metrics-server-cb6bf7c85-94hw6\" (UID: \"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218\") " pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.495463 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.495422    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:47:42.662231 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.662141    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cb6bf7c85-94hw6"]
Jul 31 23:47:42.671448 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.671406    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:42.795831 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:42.795781    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:43.071047 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:47:43.071014    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2546c4_d4bf_4d62_8bdc_11e3cdfe3218.slice/crio-842691ef856a76f8a88232ab43a4a374ed588169842ac012397e2cc40e9a416d WatchSource:0}: Error finding container 842691ef856a76f8a88232ab43a4a374ed588169842ac012397e2cc40e9a416d: Status 404 returned error can't find the container with id 842691ef856a76f8a88232ab43a4a374ed588169842ac012397e2cc40e9a416d
Jul 31 23:47:43.095477 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:43.095436    2536 generic.go:356] "Generic (PLEG): container finished" podID="9e05d46a-9ef3-4434-89d3-63db02d44a79" containerID="ad6263d3911a33ccfb64e7ac8970a07da827aa58d83e3ad60a43f01e2fa9fc3c" exitCode=0
Jul 31 23:47:43.095935 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:43.095534    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerDied","Data":"ad6263d3911a33ccfb64e7ac8970a07da827aa58d83e3ad60a43f01e2fa9fc3c"}
Jul 31 23:47:43.097083 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:43.097058    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6" event={"ID":"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218","Type":"ContainerStarted","Data":"842691ef856a76f8a88232ab43a4a374ed588169842ac012397e2cc40e9a416d"}
Jul 31 23:47:43.146684 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:43.146623    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:43.954592 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:43.954555    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.048191 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.048145    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.113050 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.113016    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"653cb4f76b148a179cb4517b04699b35d4c28419e271d2a91174b46d6751a6b6"}
Jul 31 23:47:44.113517 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.113083    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"8edd08ea90892a623bab94358f21aa17f37b666fb32513af0ca1c51338c08ffb"}
Jul 31 23:47:44.113517 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.113111    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"63eb28042988b6db9453c2d8c2ff211044d79511de520f87852a1a3d79648697"}
Jul 31 23:47:44.157529 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.157473    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.665024 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.664981    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:47:44.669914 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.669887    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:47:44.737330 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.737268    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.797100 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.797065    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.870374 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.870339    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.971828 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:44.971797    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:45.024823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.024793    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:45.121702 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.121666    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6" event={"ID":"5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218","Type":"ContainerStarted","Data":"7dd6a008cf816ed6b479606e6e1e6096592beae74391e987864ce456dcfb463e"}
Jul 31 23:47:45.125899 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.125862    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"78cfde3aaa1e0d58742938f21b9898b03705135b1c0a13998c90c45306b24400"}
Jul 31 23:47:45.126051 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.125950    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"526bd66719c5a036613d74ab7e2677581284abfca725ad4f902f3880ffaac3ff"}
Jul 31 23:47:45.129805 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.129774    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"d6a24b7b893ca90a921356d7b851061e44356b8eb51481e528f965f976aa68c5"}
Jul 31 23:47:45.138836 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:45.138780    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6" podStartSLOduration=1.279069019 podStartE2EDuration="3.138756409s" totalImagesPullingTime="1.85968739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:45.138062312 +0000 UTC m=+90.103257955" watchObservedRunningTime="2026-07-31 23:47:45.138756409 +0000 UTC m=+90.103952032"
Jul 31 23:47:46.137417 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.137375    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" event={"ID":"fd92ec36-2398-44dc-b554-2a461bfb083d","Type":"ContainerStarted","Data":"c71efc0381ba4a02dd865960a7c052b58d7ac119508e7d270aae46b482a34e14"}
Jul 31 23:47:46.137867 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.137497    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:46.140893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.140867    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"5083bb45e18ad39644eb3e0705ba8237bad6b435e4c2d9dd0672e812dc921709"}
Jul 31 23:47:46.141023 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.140923    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"8abf12e6021c05fc08d5f3712c3be6e031eed198845fc0dc123ae6dd69e1484b"}
Jul 31 23:47:46.141023 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.140950    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"4b1f82bf731c367257648cd04cc5929ca45f33b04e3b79503d3faf0b11eb59cf"}
Jul 31 23:47:46.141023 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.140979    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"72541d025dea3715ff2a82e96e8da02b4faad54d74d42adf8db8fb4a16265851"}
Jul 31 23:47:46.141023 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.141004    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e05d46a-9ef3-4434-89d3-63db02d44a79","Type":"ContainerStarted","Data":"0ddc85ae228f71f63737d72448f18272ef3e16209d0f94af2ba82f5472a088f1"}
Jul 31 23:47:46.162357 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.162297    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j" podStartSLOduration=2.818724381 podStartE2EDuration="6.162282058s" totalImagesPullingTime="3.343557677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:46.161296084 +0000 UTC m=+91.126491706" watchObservedRunningTime="2026-07-31 23:47:46.162282058 +0000 UTC m=+91.127477682"
Jul 31 23:47:46.186424 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:46.186372    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.309087432 podStartE2EDuration="8.186355007s" totalImagesPullingTime="2.858788756s" totalInitContainerRuntime="18.478819ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:46.184711114 +0000 UTC m=+91.149906737" watchObservedRunningTime="2026-07-31 23:47:46.186355007 +0000 UTC m=+91.151550629"
Jul 31 23:47:48.486823 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:48.486783    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:49.226682 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:49.226633    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:49.324474 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:49.324431    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:52.155550 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:52.155519    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8fff5d969-5p94j"
Jul 31 23:47:52.937655 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:52.937617    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8kmsg"
Jul 31 23:47:54.021317 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:54.021273    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:54.118534 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:54.118474    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:54.220014 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:54.219983    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:54.320019 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:54.319981    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:58.147389 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:47:58.147354    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:02.496308 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:02.496266    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:48:02.496861 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:02.496346    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:48:06.061853 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:06.061802    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:06.162998 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:06.162967    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:09.685972 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:09.685917    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" podUID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" containerName="registry" containerID="cri-o://96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5" gracePeriod=30
Jul 31 23:48:09.943506 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:09.943432    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:48:10.080706 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080668    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.080903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080742    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.080903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080776    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.080903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080803    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.080903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080830    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.080903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080865    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.081156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080910    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72q8n\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.081156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.080955    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca\") pod \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\" (UID: \"2eac1e71-b840-4cc3-8c3f-0a45a8595420\") "
Jul 31 23:48:10.081446 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.081381    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:10.081616 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.081589    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:10.089347 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.089303    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:10.089513 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.089376    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:10.089818 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.089768    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:10.089818 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.089781    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:10.089960 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.089890    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n" (OuterVolumeSpecName: "kube-api-access-72q8n") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "kube-api-access-72q8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:10.092926 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.092897    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2eac1e71-b840-4cc3-8c3f-0a45a8595420" (UID: "2eac1e71-b840-4cc3-8c3f-0a45a8595420"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:10.182203 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182161    2536 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-trusted-ca\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182203 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182196    2536 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182203 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182214    2536 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-installation-pull-secrets\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182225    2536 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-bound-sa-token\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182237    2536 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2eac1e71-b840-4cc3-8c3f-0a45a8595420-image-registry-private-configuration\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182246    2536 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2eac1e71-b840-4cc3-8c3f-0a45a8595420-registry-certificates\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182254    2536 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2eac1e71-b840-4cc3-8c3f-0a45a8595420-ca-trust-extracted\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.182438 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.182262    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72q8n\" (UniqueName: \"kubernetes.io/projected/2eac1e71-b840-4cc3-8c3f-0a45a8595420-kube-api-access-72q8n\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:48:10.242757 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.242662    2536 generic.go:356] "Generic (PLEG): container finished" podID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" containerID="96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5" exitCode=0
Jul 31 23:48:10.242757 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.242734    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5"
Jul 31 23:48:10.242757 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.242746    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" event={"ID":"2eac1e71-b840-4cc3-8c3f-0a45a8595420","Type":"ContainerDied","Data":"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"}
Jul 31 23:48:10.242971 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.242833    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76db5bc764-v6hj5" event={"ID":"2eac1e71-b840-4cc3-8c3f-0a45a8595420","Type":"ContainerDied","Data":"908c63b7046b6f78c71ce54c2d89e828b62233c41157686dff7b17f0c6c5650f"}
Jul 31 23:48:10.242971 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.242861    2536 scope.go:117] "RemoveContainer" containerID="96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"
Jul 31 23:48:10.251575 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.251555    2536 scope.go:117] "RemoveContainer" containerID="96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"
Jul 31 23:48:10.251828 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:48:10.251808    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5\": container with ID starting with 96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5 not found: ID does not exist" containerID="96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"
Jul 31 23:48:10.251875 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.251839    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5"} err="failed to get container status \"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5\": rpc error: code = NotFound desc = could not find container \"96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5\": container with ID starting with 96b7a2cde73214e37c0714f595024aabf203e6f533bc8abf22a9bc00b9fc82b5 not found: ID does not exist"
Jul 31 23:48:10.263606 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.263579    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:48:10.267272 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:10.267248    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76db5bc764-v6hj5"]
Jul 31 23:48:11.561411 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:11.561378    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" path="/var/lib/kubelet/pods/2eac1e71-b840-4cc3-8c3f-0a45a8595420/volumes"
Jul 31 23:48:13.851734 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:13.851695    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:13.951933 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:13.951895    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:19.280437 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:19.280399    2536 generic.go:356] "Generic (PLEG): container finished" podID="1fe52f27-2747-45aa-a384-14548caf51b6" containerID="8fa66c6978b8a9565417ae707bbd5c42ce29bdd8d7a1614d9e80507b2d11f37a" exitCode=0
Jul 31 23:48:19.280808 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:19.280461    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj" event={"ID":"1fe52f27-2747-45aa-a384-14548caf51b6","Type":"ContainerDied","Data":"8fa66c6978b8a9565417ae707bbd5c42ce29bdd8d7a1614d9e80507b2d11f37a"}
Jul 31 23:48:19.280886 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:19.280871    2536 scope.go:117] "RemoveContainer" containerID="8fa66c6978b8a9565417ae707bbd5c42ce29bdd8d7a1614d9e80507b2d11f37a"
Jul 31 23:48:20.286511 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:20.286448    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-wvjfj" event={"ID":"1fe52f27-2747-45aa-a384-14548caf51b6","Type":"ContainerStarted","Data":"c47d0d5af189ccae119cbb7e9441766264d960ae73383876650f78ab62f87497"}
Jul 31 23:48:22.500793 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:22.500767    2536 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:48:22.504512 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:22.504472    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cb6bf7c85-94hw6"
Jul 31 23:48:25.308836 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:25.308794    2536 generic.go:356] "Generic (PLEG): container finished" podID="2ad66796-3dd4-48e5-856a-d7deaf771563" containerID="6b78459469d8c53760b246a89961f3f3b1eb541d2299f58274a73016f6c950d9" exitCode=0
Jul 31 23:48:25.309235 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:25.308868    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8" event={"ID":"2ad66796-3dd4-48e5-856a-d7deaf771563","Type":"ContainerDied","Data":"6b78459469d8c53760b246a89961f3f3b1eb541d2299f58274a73016f6c950d9"}
Jul 31 23:48:25.309235 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:25.309211    2536 scope.go:117] "RemoveContainer" containerID="6b78459469d8c53760b246a89961f3f3b1eb541d2299f58274a73016f6c950d9"
Jul 31 23:48:26.314378 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:26.314345    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-vjzn8" event={"ID":"2ad66796-3dd4-48e5-856a-d7deaf771563","Type":"ContainerStarted","Data":"2211cd73efdc4dbb69e12b176dab336f5f93fe6e45e6f28c9fbed1d43ae1bf63"}
Jul 31 23:48:28.251396 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:28.251341    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:28.351956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:28.351920    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:28.452098 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:28.452053    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:29.328389 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:29.328355    2536 generic.go:356] "Generic (PLEG): container finished" podID="d7fd0abc-48ff-4224-8cf3-2bb4b369193a" containerID="793f3fbe0bf48449245a9a32f65d6aa327b99161d7593c4ac6e25d564be6d04a" exitCode=0
Jul 31 23:48:29.328754 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:29.328427    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-g2zqs" event={"ID":"d7fd0abc-48ff-4224-8cf3-2bb4b369193a","Type":"ContainerDied","Data":"793f3fbe0bf48449245a9a32f65d6aa327b99161d7593c4ac6e25d564be6d04a"}
Jul 31 23:48:29.328794 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:29.328778    2536 scope.go:117] "RemoveContainer" containerID="793f3fbe0bf48449245a9a32f65d6aa327b99161d7593c4ac6e25d564be6d04a"
Jul 31 23:48:30.334492 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:30.334441    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-g2zqs" event={"ID":"d7fd0abc-48ff-4224-8cf3-2bb4b369193a","Type":"ContainerStarted","Data":"c50d8c746bf09526baf0c9a46f7c5db89b2a316b0ec36556d0cfd3c92812bf48"}
Jul 31 23:48:48.762214 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:48.762176    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:57.059932 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:57.059853    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:57.275182 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:48:57.275141    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.156781 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.156744    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"]
Jul 31 23:49:02.157267 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.157245    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" containerName="registry"
Jul 31 23:49:02.157342 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.157271    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" containerName="registry"
Jul 31 23:49:02.157402 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.157372    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eac1e71-b840-4cc3-8c3f-0a45a8595420" containerName="registry"
Jul 31 23:49:02.162249 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.162221    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.164677 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.164648    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 31 23:49:02.164824 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.164743    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 31 23:49:02.164997 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.164978    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 31 23:49:02.165091 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.164994    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n8gv2\""
Jul 31 23:49:02.165249 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.165224    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 31 23:49:02.165392 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.165373    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 31 23:49:02.171963 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.171934    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 31 23:49:02.172882 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.172734    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"]
Jul 31 23:49:02.201062 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201025    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-federate-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201266 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201071    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201266 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201106    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201266 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201214    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201266 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201259    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-serving-certs-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201444 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201318    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6pp\" (UniqueName: \"kubernetes.io/projected/20f75156-54e4-4161-9f30-530c4a176ad0-kube-api-access-mz6pp\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201444 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201359    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-metrics-client-ca\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.201444 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.201387    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302361 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302318    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302367    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-serving-certs-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302394    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6pp\" (UniqueName: \"kubernetes.io/projected/20f75156-54e4-4161-9f30-530c4a176ad0-kube-api-access-mz6pp\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302418    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-metrics-client-ca\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302448    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302531    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-federate-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302561    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.302595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.302589    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.303245 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.303214    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-serving-certs-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.303418 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.303395    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-metrics-client-ca\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.303515 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.303438    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.305153 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.305122    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.305313 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.305289    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-secret-telemeter-client\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.305356 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.305318    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-telemeter-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.305356 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.305299    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/20f75156-54e4-4161-9f30-530c4a176ad0-federate-client-tls\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.309734 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.309717    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6pp\" (UniqueName: \"kubernetes.io/projected/20f75156-54e4-4161-9f30-530c4a176ad0-kube-api-access-mz6pp\") pod \"telemeter-client-56cc97f9d5-wwjzg\" (UID: \"20f75156-54e4-4161-9f30-530c4a176ad0\") " pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.474166 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.474066    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"
Jul 31 23:49:02.616838 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:02.616802    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg"]
Jul 31 23:49:02.619267 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:49:02.619238    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f75156_54e4_4161_9f30_530c4a176ad0.slice/crio-5e2534beef3fee9d58063a3eba71e520e6580763d800c54bd3c634dcde9a6022 WatchSource:0}: Error finding container 5e2534beef3fee9d58063a3eba71e520e6580763d800c54bd3c634dcde9a6022: Status 404 returned error can't find the container with id 5e2534beef3fee9d58063a3eba71e520e6580763d800c54bd3c634dcde9a6022
Jul 31 23:49:03.484951 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:03.484908    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg" event={"ID":"20f75156-54e4-4161-9f30-530c4a176ad0","Type":"ContainerStarted","Data":"5e2534beef3fee9d58063a3eba71e520e6580763d800c54bd3c634dcde9a6022"}
Jul 31 23:49:03.574288 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:03.574246    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:04.491204 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.491119    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg" event={"ID":"20f75156-54e4-4161-9f30-530c4a176ad0","Type":"ContainerStarted","Data":"60dd51c13922890567b29e33bef36b6344e82c18196d4545ffbc1a7ca8634c35"}
Jul 31 23:49:04.491204 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.491200    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg" event={"ID":"20f75156-54e4-4161-9f30-530c4a176ad0","Type":"ContainerStarted","Data":"94c3a1ad788b6806f8b5aad566f9b479d27652db5d02d87423678c1414e2cb1d"}
Jul 31 23:49:04.491615 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.491223    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg" event={"ID":"20f75156-54e4-4161-9f30-530c4a176ad0","Type":"ContainerStarted","Data":"b4ca64ce2109b584c18447ca196caa0dd10b7045d6b256990dedf1802caf2993"}
Jul 31 23:49:04.512866 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.512814    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56cc97f9d5-wwjzg" podStartSLOduration=0.899091257 podStartE2EDuration="2.512799474s" totalImagesPullingTime="1.613708217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:49:04.510514554 +0000 UTC m=+169.475710175" watchObservedRunningTime="2026-07-31 23:49:04.512799474 +0000 UTC m=+169.477995096"
Jul 31 23:49:04.645595 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.645563    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:04.789293 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:04.789257    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:05.408927 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:05.408871    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:05.465092 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:05.465057    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:12.269670 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:12.269585    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:12.326881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:12.326845    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:15.617419 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:15.617354    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:15.678657 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:15.678620    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:16.212930 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:16.212892    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:24.270228 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:24.270189    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:24.331784 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:24.331750    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:26.377867 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:26.377826    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:26.431556 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:26.431505    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:26.490089 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:26.490051    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:36.074746 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:36.074710    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:36.174208 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:36.174171    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:57.239336 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:49:57.239292    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.264996 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:00.264963    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:20.165246 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:20.165204    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:20.221167 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:20.221132    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:29.616298 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:29.616257    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:29.718017 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:29.717980    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:29.776163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:29.776126    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:51.590465 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:51.590382    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:51.686324 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:51.686292    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.774809 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.774768    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"]
Jul 31 23:50:52.778188 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.778165    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.780567 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.780547    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2r6cg\""
Jul 31 23:50:52.780690 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.780546    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 23:50:52.780690 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.780546    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 23:50:52.780690 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.780551    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 23:50:52.789327 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.789302    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"]
Jul 31 23:50:52.841376 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.841343    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngww\" (UniqueName: \"kubernetes.io/projected/c1e6edbf-983a-4946-8e0c-c95d53ce8960-kube-api-access-lngww\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.841566 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.841397    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1e6edbf-983a-4946-8e0c-c95d53ce8960-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.942429 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.942398    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngww\" (UniqueName: \"kubernetes.io/projected/c1e6edbf-983a-4946-8e0c-c95d53ce8960-kube-api-access-lngww\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.942623 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.942456    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1e6edbf-983a-4946-8e0c-c95d53ce8960-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.944763 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.944742    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1e6edbf-983a-4946-8e0c-c95d53ce8960-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:52.955257 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:52.955230    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngww\" (UniqueName: \"kubernetes.io/projected/c1e6edbf-983a-4946-8e0c-c95d53ce8960-kube-api-access-lngww\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-t728x\" (UID: \"c1e6edbf-983a-4946-8e0c-c95d53ce8960\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:53.089158 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:53.089121    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:53.224456 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:53.224425    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"]
Jul 31 23:50:53.227598 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:50:53.227572    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e6edbf_983a_4946_8e0c_c95d53ce8960.slice/crio-f35e6e06e21d7d410875a455944cfe83221be37a8fce94fa9c22ee022640ea5a WatchSource:0}: Error finding container f35e6e06e21d7d410875a455944cfe83221be37a8fce94fa9c22ee022640ea5a: Status 404 returned error can't find the container with id f35e6e06e21d7d410875a455944cfe83221be37a8fce94fa9c22ee022640ea5a
Jul 31 23:50:53.944933 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:53.944899    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x" event={"ID":"c1e6edbf-983a-4946-8e0c-c95d53ce8960","Type":"ContainerStarted","Data":"f35e6e06e21d7d410875a455944cfe83221be37a8fce94fa9c22ee022640ea5a"}
Jul 31 23:50:56.899709 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:56.899674    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:56.962006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:56.961968    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x" event={"ID":"c1e6edbf-983a-4946-8e0c-c95d53ce8960","Type":"ContainerStarted","Data":"3a9e12d918947ee63cd67a7124fc830aa039f109fb0251d82ef875fd104e36f7"}
Jul 31 23:50:56.962184 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:56.962096    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:50:56.981975 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:56.981922    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x" podStartSLOduration=1.73773459 podStartE2EDuration="4.981904578s" totalImagesPullingTime="3.244169988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:56.979205895 +0000 UTC m=+281.944401518" watchObservedRunningTime="2026-07-31 23:50:56.981904578 +0000 UTC m=+281.947100201"
Jul 31 23:50:57.172480 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.172385    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.375703 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.375664    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.527479 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.527392    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.803084 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.803046    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.975059 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.975023    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6p697"]
Jul 31 23:50:57.977545 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.977512    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.978301 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.978284    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:57.980673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.980646    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 31 23:50:57.980673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.980667    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hvqdd\""
Jul 31 23:50:57.980804 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.980678    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 23:50:57.987674 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:57.987639    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6p697"]
Jul 31 23:50:58.050320 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.050280    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.090556 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.090450    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rs6\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-kube-api-access-47rs6\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.090757 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.090717    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.150626 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.150586    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.191993 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.191957    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.192205 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.192086    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rs6\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-kube-api-access-47rs6\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.192205 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:50:58.192137    2536 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:58.192205 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:50:58.192166    2536 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-6p697: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:58.192360 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:50:58.192237    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates podName:0573f5d2-fe4f-41f0-83e4-e04c71f3a538 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:58.692215853 +0000 UTC m=+283.657411456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates") pod "keda-admission-56f9798bb4-6p697" (UID: "0573f5d2-fe4f-41f0-83e4-e04c71f3a538") : secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:58.203259 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.202377    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rs6\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-kube-api-access-47rs6\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.251066 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.251033    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.350042 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.349934    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.450759 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.450726    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.651784 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.651675    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.697007 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.696960    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.699417 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.699384    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0573f5d2-fe4f-41f0-83e4-e04c71f3a538-certificates\") pod \"keda-admission-56f9798bb4-6p697\" (UID: \"0573f5d2-fe4f-41f0-83e4-e04c71f3a538\") " pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:58.711808 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.711776    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:58.889813 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:58.889778    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:50:59.021811 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:59.021786    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6p697"]
Jul 31 23:50:59.024459 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:50:59.024433    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0573f5d2_fe4f_41f0_83e4_e04c71f3a538.slice/crio-cb6458ec5bfe56c0b80de0ef74139d1b1f7fdf885f5a4a4d4e4e1d98dff1cb25 WatchSource:0}: Error finding container cb6458ec5bfe56c0b80de0ef74139d1b1f7fdf885f5a4a4d4e4e1d98dff1cb25: Status 404 returned error can't find the container with id cb6458ec5bfe56c0b80de0ef74139d1b1f7fdf885f5a4a4d4e4e1d98dff1cb25
Jul 31 23:50:59.977137 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:50:59.977095    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-6p697" event={"ID":"0573f5d2-fe4f-41f0-83e4-e04c71f3a538","Type":"ContainerStarted","Data":"cb6458ec5bfe56c0b80de0ef74139d1b1f7fdf885f5a4a4d4e4e1d98dff1cb25"}
Jul 31 23:51:00.982973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:00.982935    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-6p697" event={"ID":"0573f5d2-fe4f-41f0-83e4-e04c71f3a538","Type":"ContainerStarted","Data":"701a394c0c82199454d15477de7054d6f04e5c73bb71080255b19354b6add962"}
Jul 31 23:51:00.983373 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:00.983027    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:51:00.998999 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:00.998952    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-6p697" podStartSLOduration=2.745200665 podStartE2EDuration="3.998937806s" totalImagesPullingTime="1.253737141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:00.997349733 +0000 UTC m=+285.962545358" watchObservedRunningTime="2026-07-31 23:51:00.998937806 +0000 UTC m=+285.964133454"
Jul 31 23:51:15.466911 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:15.466883    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:51:15.468697 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:15.468675    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:51:15.480903 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:15.480880    2536 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:17.968361 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:17.968332    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-t728x"
Jul 31 23:51:21.991047 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:21.991016    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-6p697"
Jul 31 23:51:50.426048 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:50.426010    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:51.583621 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:51.583585    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:56.003629 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:56.003595    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:56.095732 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:56.095690    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:57.963280 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:57.963237    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:58.037335 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:51:58.037292    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.331058 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.330946    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.398318 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.398272    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:15.400597 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.400566    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.401858 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.401840    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.404753 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.404727    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:52:15.404935 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.404767    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 23:52:15.404935 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.404777    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-jrt9h\""
Jul 31 23:52:15.405638 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.405611    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:52:15.413498 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.413448    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:15.425097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.425055    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-mdgp4"]
Jul 31 23:52:15.429228 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.429194    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.431673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.431640    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 23:52:15.431846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.431676    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5dcjg\""
Jul 31 23:52:15.437655 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.437623    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-mdgp4"]
Jul 31 23:52:15.475403 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.475367    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlc7\" (UniqueName: \"kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.475403 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.475413    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.475705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.475470    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-data\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.475705 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.475578    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvkr\" (UniqueName: \"kubernetes.io/projected/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-kube-api-access-6qvkr\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.488565 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.488523    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.576770 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.576714    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-data\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.576993 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.576804    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvkr\" (UniqueName: \"kubernetes.io/projected/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-kube-api-access-6qvkr\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.576993 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.576872    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlc7\" (UniqueName: \"kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.576993 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.576899    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.577228 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.577189    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-data\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.579294 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.579268    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 23:52:15.587251 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.587182    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:52:15.589929 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.589903    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.591255 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.591227    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.598363 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.598337    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:52:15.608559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.608530    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlc7\" (UniqueName: \"kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7\") pod \"kserve-controller-manager-674d959fb8-7pn95\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") " pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.608559 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.608550    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvkr\" (UniqueName: \"kubernetes.io/projected/dff8f2aa-2b39-45f9-b725-0ab1e36f443e-kube-api-access-6qvkr\") pod \"seaweedfs-64568bcd49-mdgp4\" (UID: \"dff8f2aa-2b39-45f9-b725-0ab1e36f443e\") " pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.715911 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.715875    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-jrt9h\""
Jul 31 23:52:15.723815 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.723780    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:15.742074 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.742031    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:15.775480 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.775440    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.888752 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.888718    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:15.890137 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:52:15.890106    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2156e01_ac7c_43fe_a241_c5cc86c472de.slice/crio-62cdcec55106bdd0d798320d9ce57a07e937242c1643c26949c03a5abde950ff WatchSource:0}: Error finding container 62cdcec55106bdd0d798320d9ce57a07e937242c1643c26949c03a5abde950ff: Status 404 returned error can't find the container with id 62cdcec55106bdd0d798320d9ce57a07e937242c1643c26949c03a5abde950ff
Jul 31 23:52:15.892210 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.892181    2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:52:15.917938 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.917911    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-mdgp4"]
Jul 31 23:52:15.920632 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:52:15.920601    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff8f2aa_2b39_45f9_b725_0ab1e36f443e.slice/crio-be13f5ac2e402ff514abf7bd50e3975ccc8b0641f532d29de19d6c97901118c6 WatchSource:0}: Error finding container be13f5ac2e402ff514abf7bd50e3975ccc8b0641f532d29de19d6c97901118c6: Status 404 returned error can't find the container with id be13f5ac2e402ff514abf7bd50e3975ccc8b0641f532d29de19d6c97901118c6
Jul 31 23:52:15.946194 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:15.946148    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:16.048044 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:16.048001    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:16.148658 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:16.148551    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:16.321936 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:16.321893    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-mdgp4" event={"ID":"dff8f2aa-2b39-45f9-b725-0ab1e36f443e","Type":"ContainerStarted","Data":"be13f5ac2e402ff514abf7bd50e3975ccc8b0641f532d29de19d6c97901118c6"}
Jul 31 23:52:16.324451 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:16.324377    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" event={"ID":"b2156e01-ac7c-43fe-a241-c5cc86c472de","Type":"ContainerStarted","Data":"62cdcec55106bdd0d798320d9ce57a07e937242c1643c26949c03a5abde950ff"}
Jul 31 23:52:20.347014 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.346970    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-mdgp4" event={"ID":"dff8f2aa-2b39-45f9-b725-0ab1e36f443e","Type":"ContainerStarted","Data":"dfd355dee856190e4551241d8d6b27f4b20cfe1f049d2827282ab400e9179dbe"}
Jul 31 23:52:20.347466 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.347088    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:20.348724 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.348698    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" event={"ID":"b2156e01-ac7c-43fe-a241-c5cc86c472de","Type":"ContainerStarted","Data":"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"}
Jul 31 23:52:20.348854 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.348762    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:20.363923 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.363875    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-mdgp4" podStartSLOduration=1.7424327609999999 podStartE2EDuration="5.363856235s" totalImagesPullingTime="3.621423474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:20.362202592 +0000 UTC m=+365.327398228" watchObservedRunningTime="2026-07-31 23:52:20.363856235 +0000 UTC m=+365.329051857"
Jul 31 23:52:20.378579 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:20.378516    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" podStartSLOduration=1.857130373 podStartE2EDuration="5.378495497s" totalImagesPullingTime="3.521365124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:20.376868307 +0000 UTC m=+365.342063933" watchObservedRunningTime="2026-07-31 23:52:20.378495497 +0000 UTC m=+365.343691111"
Jul 31 23:52:26.355828 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:26.355793    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-mdgp4"
Jul 31 23:52:32.177426 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:32.177389    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:32.237708 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:32.237658    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:43.023714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:43.023673    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:43.125282 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:43.125240    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:43.226260 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:43.226212    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:51.358545 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:51.358507    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:57.534100 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.534058    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:57.534745 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.534335    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" podUID="b2156e01-ac7c-43fe-a241-c5cc86c472de" containerName="manager" containerID="cri-o://cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc" gracePeriod=10
Jul 31 23:52:57.562423 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.562390    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-r8hst"]
Jul 31 23:52:57.564782 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.564763    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.568684 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.568652    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-r8hst"]
Jul 31 23:52:57.655249 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.655204    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14430194-c798-4faa-a171-f4d4b8515152-cert\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.655439 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.655265    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5s65\" (UniqueName: \"kubernetes.io/projected/14430194-c798-4faa-a171-f4d4b8515152-kube-api-access-v5s65\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.755793 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.755752    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14430194-c798-4faa-a171-f4d4b8515152-cert\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.756052 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.755823    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5s65\" (UniqueName: \"kubernetes.io/projected/14430194-c798-4faa-a171-f4d4b8515152-kube-api-access-v5s65\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.758257 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.758220    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14430194-c798-4faa-a171-f4d4b8515152-cert\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.765623 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.765585    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5s65\" (UniqueName: \"kubernetes.io/projected/14430194-c798-4faa-a171-f4d4b8515152-kube-api-access-v5s65\") pod \"kserve-controller-manager-674d959fb8-r8hst\" (UID: \"14430194-c798-4faa-a171-f4d4b8515152\") " pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.790236 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.790156    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:57.856726 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.856692    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert\") pod \"b2156e01-ac7c-43fe-a241-c5cc86c472de\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") "
Jul 31 23:52:57.856914 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.856751    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdlc7\" (UniqueName: \"kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7\") pod \"b2156e01-ac7c-43fe-a241-c5cc86c472de\" (UID: \"b2156e01-ac7c-43fe-a241-c5cc86c472de\") "
Jul 31 23:52:57.858951 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.858917    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert" (OuterVolumeSpecName: "cert") pod "b2156e01-ac7c-43fe-a241-c5cc86c472de" (UID: "b2156e01-ac7c-43fe-a241-c5cc86c472de"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:52:57.859066 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.858968    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7" (OuterVolumeSpecName: "kube-api-access-mdlc7") pod "b2156e01-ac7c-43fe-a241-c5cc86c472de" (UID: "b2156e01-ac7c-43fe-a241-c5cc86c472de"). InnerVolumeSpecName "kube-api-access-mdlc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:57.926476 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.926440    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:57.957810 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.957766    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlc7\" (UniqueName: \"kubernetes.io/projected/b2156e01-ac7c-43fe-a241-c5cc86c472de-kube-api-access-mdlc7\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:52:57.957810 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:57.957801    2536 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2156e01-ac7c-43fe-a241-c5cc86c472de-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:52:58.058319 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.058295    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-r8hst"]
Jul 31 23:52:58.060912 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:52:58.060883    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14430194_c798_4faa_a171_f4d4b8515152.slice/crio-8d31856c796dfc2428129e3a08ca4ae280ab6a1b4ef6973be4a127eb61f79c8d WatchSource:0}: Error finding container 8d31856c796dfc2428129e3a08ca4ae280ab6a1b4ef6973be4a127eb61f79c8d: Status 404 returned error can't find the container with id 8d31856c796dfc2428129e3a08ca4ae280ab6a1b4ef6973be4a127eb61f79c8d
Jul 31 23:52:58.520775 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.520734    2536 generic.go:356] "Generic (PLEG): container finished" podID="b2156e01-ac7c-43fe-a241-c5cc86c472de" containerID="cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc" exitCode=0
Jul 31 23:52:58.520986 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.520792    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-7pn95"
Jul 31 23:52:58.520986 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.520821    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" event={"ID":"b2156e01-ac7c-43fe-a241-c5cc86c472de","Type":"ContainerDied","Data":"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"}
Jul 31 23:52:58.520986 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.520882    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-7pn95" event={"ID":"b2156e01-ac7c-43fe-a241-c5cc86c472de","Type":"ContainerDied","Data":"62cdcec55106bdd0d798320d9ce57a07e937242c1643c26949c03a5abde950ff"}
Jul 31 23:52:58.520986 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.520913    2536 scope.go:117] "RemoveContainer" containerID="cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"
Jul 31 23:52:58.522895 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.522867    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-r8hst" event={"ID":"14430194-c798-4faa-a171-f4d4b8515152","Type":"ContainerStarted","Data":"83eec57c084eff1cea7fee86247510f1972f49a6d123c895859268662cae48ea"}
Jul 31 23:52:58.523029 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.522925    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-r8hst" event={"ID":"14430194-c798-4faa-a171-f4d4b8515152","Type":"ContainerStarted","Data":"8d31856c796dfc2428129e3a08ca4ae280ab6a1b4ef6973be4a127eb61f79c8d"}
Jul 31 23:52:58.523029 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.522957    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:52:58.530833 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.530812    2536 scope.go:117] "RemoveContainer" containerID="cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"
Jul 31 23:52:58.531150 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:52:58.531129    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc\": container with ID starting with cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc not found: ID does not exist" containerID="cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"
Jul 31 23:52:58.531200 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.531161    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc"} err="failed to get container status \"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc\": rpc error: code = NotFound desc = could not find container \"cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc\": container with ID starting with cc8573a228fd40c6ebfddfd332322eeb8cb642dcc1191bf7a3f0e579e6dd74cc not found: ID does not exist"
Jul 31 23:52:58.542285 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.542236    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-r8hst" podStartSLOduration=1.252731577 podStartE2EDuration="1.542219776s" totalImagesPullingTime="289.488199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:58.540318583 +0000 UTC m=+403.505514222" watchObservedRunningTime="2026-07-31 23:52:58.542219776 +0000 UTC m=+403.507415430"
Jul 31 23:52:58.555190 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.555158    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:58.559003 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:58.558977    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-7pn95"]
Jul 31 23:52:59.567391 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:52:59.567352    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2156e01-ac7c-43fe-a241-c5cc86c472de" path="/var/lib/kubelet/pods/b2156e01-ac7c-43fe-a241-c5cc86c472de/volumes"
Jul 31 23:53:29.535249 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:29.535212    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-r8hst"
Jul 31 23:53:30.514996 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.514960    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:30.571787 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.571749    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-27kk2"]
Jul 31 23:53:30.572207 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.572143    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2156e01-ac7c-43fe-a241-c5cc86c472de" containerName="manager"
Jul 31 23:53:30.572207 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.572156    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2156e01-ac7c-43fe-a241-c5cc86c472de" containerName="manager"
Jul 31 23:53:30.572292 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.572214    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2156e01-ac7c-43fe-a241-c5cc86c472de" containerName="manager"
Jul 31 23:53:30.575700 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.575672    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.579168 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.579138    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6qgxm\""
Jul 31 23:53:30.579332 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.579138    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 23:53:30.579332 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.579145    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 23:53:30.586320 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.586288    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-27kk2"]
Jul 31 23:53:30.599938 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.599897    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:30.663601 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.663559    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:30.748366 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.748333    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl2t5\" (UniqueName: \"kubernetes.io/projected/fca21df4-4003-4197-baaa-188e448a3e32-kube-api-access-gl2t5\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.748576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.748399    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-metrics-certs\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.748576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.748438    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-cert\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.848881 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.848843    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-cert\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.849061 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.848914    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl2t5\" (UniqueName: \"kubernetes.io/projected/fca21df4-4003-4197-baaa-188e448a3e32-kube-api-access-gl2t5\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.849061 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.848969    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-metrics-certs\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.851544 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.851480    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-metrics-certs\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.851690 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.851669    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca21df4-4003-4197-baaa-188e448a3e32-cert\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.856942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.856914    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl2t5\" (UniqueName: \"kubernetes.io/projected/fca21df4-4003-4197-baaa-188e448a3e32-kube-api-access-gl2t5\") pod \"odh-model-controller-8588677cfb-27kk2\" (UID: \"fca21df4-4003-4197-baaa-188e448a3e32\") " pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.888031 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.887991    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:30.996685 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:30.996467    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:31.025074 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:31.025001    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-27kk2"]
Jul 31 23:53:31.027980 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:53:31.027948    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca21df4_4003_4197_baaa_188e448a3e32.slice/crio-362791ef178eed711a246efcfd48ad48b512c9f2244887542e461f0843886c0e WatchSource:0}: Error finding container 362791ef178eed711a246efcfd48ad48b512c9f2244887542e461f0843886c0e: Status 404 returned error can't find the container with id 362791ef178eed711a246efcfd48ad48b512c9f2244887542e461f0843886c0e
Jul 31 23:53:31.096149 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:31.096110    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:31.152693 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:31.152610    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:31.682305 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:31.682263    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-27kk2" event={"ID":"fca21df4-4003-4197-baaa-188e448a3e32","Type":"ContainerStarted","Data":"362791ef178eed711a246efcfd48ad48b512c9f2244887542e461f0843886c0e"}
Jul 31 23:53:33.694150 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:33.694054    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-27kk2" event={"ID":"fca21df4-4003-4197-baaa-188e448a3e32","Type":"ContainerStarted","Data":"40c8e44c1f40374971ee7c128009f0e84ebeb63a65c3075643e17c3cc38e9604"}
Jul 31 23:53:33.694625 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:33.694159    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:53:33.711429 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:33.711357    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-27kk2" podStartSLOduration=1.400705799 podStartE2EDuration="3.711332531s" totalImagesPullingTime="2.310626732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:33.709559752 +0000 UTC m=+438.674755384" watchObservedRunningTime="2026-07-31 23:53:33.711332531 +0000 UTC m=+438.676528155"
Jul 31 23:53:44.701874 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:53:44.701785    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-27kk2"
Jul 31 23:54:42.565466 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:54:42.565423    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:51.518108 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:54:51.518068    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:03.858771 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:03.858733    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:03.916114 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:03.916068    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.086226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.086135    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.279942 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.279907    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.343189 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.343094    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:55:15.347005 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.346975    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.349649 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349616    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 31 23:55:15.349649 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349620    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:55:15.349850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349702    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 23:55:15.349850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349715    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 31 23:55:15.349850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349758    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:55:15.350036 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.349920    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wctjk\""
Jul 31 23:55:15.350817 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.350798    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:55:15.357084 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.356826    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.357084 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.356917    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:55:15.468941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.468902    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.496739 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.496698    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.496937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.496767    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqf7\" (UniqueName: \"kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.496937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.496833    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.496937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.496858    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.496937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.496882    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.568732 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.568698    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.597961 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.597856    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.597961 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.597897    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.597961 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.597916    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.597961 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.597959    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.598344 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.598001    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqf7\" (UniqueName: \"kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.598447 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.598427    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.601097 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.601072    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 31 23:55:15.601182 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.601079    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:55:15.601182 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.601088    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 31 23:55:15.606140 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.606116    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:55:15.609317 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.609291    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.609431 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.609302    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.611105 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.611079    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.617085 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.617062    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:55:15.628118 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.628088    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqf7\" (UniqueName: \"kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.663167 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.663138    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wctjk\""
Jul 31 23:55:15.671129 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.671108    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:15.771011 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.770686    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:15.811596 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:15.811564    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:55:15.813298 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:55:15.813261    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb03d4_5bab_4c65_b7db_f4e148845009.slice/crio-c563c36bceb6b5b56fa4e8ff87967514287f2d73cbfaea275494ea31141d49b1 WatchSource:0}: Error finding container c563c36bceb6b5b56fa4e8ff87967514287f2d73cbfaea275494ea31141d49b1: Status 404 returned error can't find the container with id c563c36bceb6b5b56fa4e8ff87967514287f2d73cbfaea275494ea31141d49b1
Jul 31 23:55:16.177457 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:16.177417    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerStarted","Data":"c563c36bceb6b5b56fa4e8ff87967514287f2d73cbfaea275494ea31141d49b1"}
Jul 31 23:55:19.011209 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:19.011183    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 23:55:19.196216 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:19.196174    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerStarted","Data":"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"}
Jul 31 23:55:23.218360 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:23.218273    2536 generic.go:356] "Generic (PLEG): container finished" podID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerID="4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33" exitCode=0
Jul 31 23:55:23.218360 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:23.218341    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerDied","Data":"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"}
Jul 31 23:55:37.349174 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:37.349076    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerStarted","Data":"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"}
Jul 31 23:55:39.368855 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:39.368815    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerStarted","Data":"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"}
Jul 31 23:55:39.369323 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:39.368912    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:39.369323 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:39.369048    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:39.370325 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:39.370300    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:55:39.388361 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:39.388308    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podStartSLOduration=1.9772336080000001 podStartE2EDuration="24.388291895s" totalImagesPullingTime="18.691837574s" totalInitContainerRuntime="3.719220713s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:39.387158779 +0000 UTC m=+564.352354403" watchObservedRunningTime="2026-07-31 23:55:39.388291895 +0000 UTC m=+564.353487516"
Jul 31 23:55:40.374179 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:40.374135    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:55:45.378540 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:45.378476    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:55:45.379059 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:45.379015    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:55:55.379161 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:55:55.379112    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:05.379679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:05.379635    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:15.379600 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:15.379556    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:15.503504 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:15.503464    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:56:15.504580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:15.504557    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Jul 31 23:56:25.379770 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:25.379726    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:35.379181 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:35.379126    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:45.379422 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:45.379333    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:56:55.380466 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:56:55.380431    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:57:00.083153 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.083113    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:57:00.083583 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.083345    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" containerID="cri-o://6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3" gracePeriod=30
Jul 31 23:57:00.083583 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.083414    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy" containerID="cri-o://8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f" gracePeriod=30
Jul 31 23:57:00.140304 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.140261    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:00.158792 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.158754    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:57:00.162879 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.162841    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.164973 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.164946    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 31 23:57:00.170790 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.170757    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:57:00.251774 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.251731    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.251994 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.251789    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.251994 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.251924    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76xk\" (UniqueName: \"kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.251994 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.251978    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.252283 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.252018    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.352811 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.352712    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76xk\" (UniqueName: \"kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.352811 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.352761    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.352811 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.352792    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.353263 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.352842    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.353263 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.352886    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.353768 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.353743    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.353948 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.353915    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.354112 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.354089    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.356018 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.355993    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.366822 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.366789    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76xk\" (UniqueName: \"kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk\") pod \"isvc-canary-rollback-predictor-8589bf8f95-q84jw\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.374790 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.374749    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 31 23:57:00.472552 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.472511    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:57:00.475874 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.475844    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:00.477696 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.477671    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.479799 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.479779    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4da04-predictor-serving-cert\""
Jul 31 23:57:00.479901 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.479809    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-4da04-kube-rbac-proxy-sar-config\""
Jul 31 23:57:00.488117 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.488072    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:57:00.555009 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.554963    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.555159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.555028    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.555231 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.555156    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmkj\" (UniqueName: \"kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.555231 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.555208    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.555368 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.555334    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.645147 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.645114    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:57:00.647397 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:57:00.647367    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd22fd29_0892_4089_8b0c_31018be655a2.slice/crio-639bfac229710a0ec4bb628cbee853bdc8cce8300f5320cfd98ac0c5c304537f WatchSource:0}: Error finding container 639bfac229710a0ec4bb628cbee853bdc8cce8300f5320cfd98ac0c5c304537f: Status 404 returned error can't find the container with id 639bfac229710a0ec4bb628cbee853bdc8cce8300f5320cfd98ac0c5c304537f
Jul 31 23:57:00.655857 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.655832    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.655929 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.655868    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.655929 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.655916    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmkj\" (UniqueName: \"kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.656004 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.655938    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.656046 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.656005    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.656475 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.656451    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.656770 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.656748    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.656846 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.656793    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.658467 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.658440    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.663717 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.663692    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmkj\" (UniqueName: \"kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj\") pod \"isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.776847 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.776797    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerStarted","Data":"a69fae96a874e10af0e718f7a1e8591d5f792ae2b9595d8f3c6e09fe7c8b4f61"}
Jul 31 23:57:00.777064 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.776883    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerStarted","Data":"639bfac229710a0ec4bb628cbee853bdc8cce8300f5320cfd98ac0c5c304537f"}
Jul 31 23:57:00.779306 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.779276    2536 generic.go:356] "Generic (PLEG): container finished" podID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerID="8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f" exitCode=2
Jul 31 23:57:00.779457 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.779314    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerDied","Data":"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"}
Jul 31 23:57:00.814624 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.814578    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:00.974127 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:00.974094    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:57:00.976536 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:57:00.976507    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319b6709_a459_49b6_9885_a87c4933e97f.slice/crio-978e28b83bc19b6c6f2bc6e281e58d123648248568b9c8203a557bbad6c9eca3 WatchSource:0}: Error finding container 978e28b83bc19b6c6f2bc6e281e58d123648248568b9c8203a557bbad6c9eca3: Status 404 returned error can't find the container with id 978e28b83bc19b6c6f2bc6e281e58d123648248568b9c8203a557bbad6c9eca3
Jul 31 23:57:01.365352 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:01.365309    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:01.786689 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:01.786649    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerStarted","Data":"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"}
Jul 31 23:57:01.786865 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:01.786726    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerStarted","Data":"978e28b83bc19b6c6f2bc6e281e58d123648248568b9c8203a557bbad6c9eca3"}
Jul 31 23:57:04.804985 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:04.804943    2536 generic.go:356] "Generic (PLEG): container finished" podID="fd22fd29-0892-4089-8b0c-31018be655a2" containerID="a69fae96a874e10af0e718f7a1e8591d5f792ae2b9595d8f3c6e09fe7c8b4f61" exitCode=0
Jul 31 23:57:04.805319 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:04.805006    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerDied","Data":"a69fae96a874e10af0e718f7a1e8591d5f792ae2b9595d8f3c6e09fe7c8b4f61"}
Jul 31 23:57:05.375258 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.375219    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 31 23:57:05.379645 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.379610    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 31 23:57:05.528453 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.528421    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:57:05.604276 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604190    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvqf7\" (UniqueName: \"kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7\") pod \"98bb03d4-5bab-4c65-b7db-f4e148845009\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") "
Jul 31 23:57:05.604276 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604240    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config\") pod \"98bb03d4-5bab-4c65-b7db-f4e148845009\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") "
Jul 31 23:57:05.604532 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604282    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert\") pod \"98bb03d4-5bab-4c65-b7db-f4e148845009\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") "
Jul 31 23:57:05.604532 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604427    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls\") pod \"98bb03d4-5bab-4c65-b7db-f4e148845009\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") "
Jul 31 23:57:05.604532 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604522    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location\") pod \"98bb03d4-5bab-4c65-b7db-f4e148845009\" (UID: \"98bb03d4-5bab-4c65-b7db-f4e148845009\") "
Jul 31 23:57:05.604679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604643    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98bb03d4-5bab-4c65-b7db-f4e148845009" (UID: "98bb03d4-5bab-4c65-b7db-f4e148845009"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:57:05.604679 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604653    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98bb03d4-5bab-4c65-b7db-f4e148845009" (UID: "98bb03d4-5bab-4c65-b7db-f4e148845009"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:57:05.604928 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.604907    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98bb03d4-5bab-4c65-b7db-f4e148845009" (UID: "98bb03d4-5bab-4c65-b7db-f4e148845009"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:57:05.605034 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.605020    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:57:05.605088 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.605039    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03d4-5bab-4c65-b7db-f4e148845009-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:57:05.605088 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.605054    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98bb03d4-5bab-4c65-b7db-f4e148845009-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:57:05.606591 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.606568    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7" (OuterVolumeSpecName: "kube-api-access-tvqf7") pod "98bb03d4-5bab-4c65-b7db-f4e148845009" (UID: "98bb03d4-5bab-4c65-b7db-f4e148845009"). InnerVolumeSpecName "kube-api-access-tvqf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:57:05.606651 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.606610    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98bb03d4-5bab-4c65-b7db-f4e148845009" (UID: "98bb03d4-5bab-4c65-b7db-f4e148845009"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:57:05.705661 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.705625    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98bb03d4-5bab-4c65-b7db-f4e148845009-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:57:05.705661 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.705654    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqf7\" (UniqueName: \"kubernetes.io/projected/98bb03d4-5bab-4c65-b7db-f4e148845009-kube-api-access-tvqf7\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:57:05.812125 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.812092    2536 generic.go:356] "Generic (PLEG): container finished" podID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerID="6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3" exitCode=0
Jul 31 23:57:05.812576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.812150    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"
Jul 31 23:57:05.812576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.812168    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerDied","Data":"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"}
Jul 31 23:57:05.812576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.812235    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh" event={"ID":"98bb03d4-5bab-4c65-b7db-f4e148845009","Type":"ContainerDied","Data":"c563c36bceb6b5b56fa4e8ff87967514287f2d73cbfaea275494ea31141d49b1"}
Jul 31 23:57:05.812576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.812264    2536 scope.go:117] "RemoveContainer" containerID="8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"
Jul 31 23:57:05.813892 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.813870    2536 generic.go:356] "Generic (PLEG): container finished" podID="319b6709-a459-49b6-9885-a87c4933e97f" containerID="6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c" exitCode=0
Jul 31 23:57:05.813987 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.813965    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerDied","Data":"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"}
Jul 31 23:57:05.816560 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.816525    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerStarted","Data":"612c42436db36c1103fe30f65dad00a76af1fa5563574a4fd674c844c2342d70"}
Jul 31 23:57:05.816648 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.816588    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerStarted","Data":"bbc5c6c98545d0d663c6ec50ba9ec9f6f78e9d849c27dcf4479ad380eef1be92"}
Jul 31 23:57:05.816937 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.816910    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:05.817019 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.816967    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:05.818365 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.818339    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:05.823006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.822992    2536 scope.go:117] "RemoveContainer" containerID="6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"
Jul 31 23:57:05.831678 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.831657    2536 scope.go:117] "RemoveContainer" containerID="4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"
Jul 31 23:57:05.841360 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.841340    2536 scope.go:117] "RemoveContainer" containerID="8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"
Jul 31 23:57:05.841745 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:57:05.841714    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f\": container with ID starting with 8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f not found: ID does not exist" containerID="8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"
Jul 31 23:57:05.841849 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.841756    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f"} err="failed to get container status \"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f\": rpc error: code = NotFound desc = could not find container \"8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f\": container with ID starting with 8e7381af04e45796fc64c78745d24bc93dd6694bfe6634ee6aa71012d1810a6f not found: ID does not exist"
Jul 31 23:57:05.841849 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.841781    2536 scope.go:117] "RemoveContainer" containerID="6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"
Jul 31 23:57:05.842081 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:57:05.842057    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3\": container with ID starting with 6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3 not found: ID does not exist" containerID="6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"
Jul 31 23:57:05.842144 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.842086    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3"} err="failed to get container status \"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3\": rpc error: code = NotFound desc = could not find container \"6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3\": container with ID starting with 6f8adf0aac0eefdb56bdee42463f91d5b14d9a277d155dd8b370176612fd26d3 not found: ID does not exist"
Jul 31 23:57:05.842144 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.842106    2536 scope.go:117] "RemoveContainer" containerID="4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"
Jul 31 23:57:05.842421 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:57:05.842402    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33\": container with ID starting with 4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33 not found: ID does not exist" containerID="4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"
Jul 31 23:57:05.842468 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.842432    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33"} err="failed to get container status \"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33\": rpc error: code = NotFound desc = could not find container \"4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33\": container with ID starting with 4bce2b3a77cf29f02728985ee56f05abb80c9c7ad3eab55b3a802f37c5374d33 not found: ID does not exist"
Jul 31 23:57:05.851418 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.851370    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podStartSLOduration=1.900418296 podStartE2EDuration="5.85135249s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950934194s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:05.850832289 +0000 UTC m=+650.816027912" watchObservedRunningTime="2026-07-31 23:57:05.85135249 +0000 UTC m=+650.816548124"
Jul 31 23:57:05.863128 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.863096    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:57:05.866719 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:05.866690    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-lj9gh"]
Jul 31 23:57:06.826428 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:06.826381    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:07.563908 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:07.563867    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" path="/var/lib/kubelet/pods/98bb03d4-5bab-4c65-b7db-f4e148845009/volumes"
Jul 31 23:57:11.831888 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:11.831853    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:57:11.832623 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:11.832585    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:21.833205 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:21.833157    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:23.928059 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.928022    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerStarted","Data":"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"}
Jul 31 23:57:23.928524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.928086    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerStarted","Data":"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"}
Jul 31 23:57:23.928524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.928389    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:23.928524 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.928451    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:23.929777 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.929748    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:57:23.945747 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:23.945694    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podStartSLOduration=2.715232244 podStartE2EDuration="23.945676966s" totalImagesPullingTime="17.478058919s" totalInitContainerRuntime="3.752385803s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:23.944832555 +0000 UTC m=+668.910028180" watchObservedRunningTime="2026-07-31 23:57:23.945676966 +0000 UTC m=+668.910872589"
Jul 31 23:57:24.933046 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:24.932998    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:57:29.937805 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:29.937773    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:57:29.938364 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:29.938331    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:57:31.425213 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:31.425178    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:31.833526 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:31.833460    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:38.164435 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:38.164399    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:38.218472 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:38.218431    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:39.938413 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:39.938365    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:57:41.832850 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:41.832808    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:49.938610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:49.938564    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:57:51.833186 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:51.833144    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:57:59.938862 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:57:59.938817    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:58:01.833012 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:01.832970    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:58:09.606068 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:09.605946    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:09.706735 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:09.706697    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:09.938656 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:09.938561    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:58:11.832580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:11.832526    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:58:19.938750 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:19.938709    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 31 23:58:21.833337 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:21.833289    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:58:29.938666 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:29.938634    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:58:31.833680 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:31.833648    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:58:40.241159 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.241118    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:58:40.241689 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.241394    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" containerID="cri-o://bbc5c6c98545d0d663c6ec50ba9ec9f6f78e9d849c27dcf4479ad380eef1be92" gracePeriod=30
Jul 31 23:58:40.241689 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.241596    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kube-rbac-proxy" containerID="cri-o://612c42436db36c1103fe30f65dad00a76af1fa5563574a4fd674c844c2342d70" gracePeriod=30
Jul 31 23:58:40.266892 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.266855    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:40.373678 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.373636    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:40.525991 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:40.525906    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:41.312313 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:41.312277    2536 generic.go:356] "Generic (PLEG): container finished" podID="fd22fd29-0892-4089-8b0c-31018be655a2" containerID="612c42436db36c1103fe30f65dad00a76af1fa5563574a4fd674c844c2342d70" exitCode=2
Jul 31 23:58:41.312313 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:41.312324    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerDied","Data":"612c42436db36c1103fe30f65dad00a76af1fa5563574a4fd674c844c2342d70"}
Jul 31 23:58:41.827494 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:41.827452    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 31 23:58:41.832910 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:41.832872    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 31 23:58:45.337394 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.337084    2536 generic.go:356] "Generic (PLEG): container finished" podID="fd22fd29-0892-4089-8b0c-31018be655a2" containerID="bbc5c6c98545d0d663c6ec50ba9ec9f6f78e9d849c27dcf4479ad380eef1be92" exitCode=0
Jul 31 23:58:45.337394 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.337275    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerDied","Data":"bbc5c6c98545d0d663c6ec50ba9ec9f6f78e9d849c27dcf4479ad380eef1be92"}
Jul 31 23:58:45.376993 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.376952    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:45.412875 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.412848    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:58:45.444417 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.444378    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:45.448436 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448405    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Jul 31 23:58:45.448890 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448870    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container"
Jul 31 23:58:45.448890 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448890    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448902    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448909    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448926    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="storage-initializer"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448934    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="storage-initializer"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448945    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="storage-initializer"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448953    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="storage-initializer"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448968    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448977    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448987    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449020 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.448994    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.449091    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.449106    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kube-rbac-proxy"
Jul 31 23:58:45.449536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.449115    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" containerName="kserve-container"
Jul 31 23:58:45.449536 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.449129    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="98bb03d4-5bab-4c65-b7db-f4e148845009" containerName="kserve-container"
Jul 31 23:58:45.452746 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.452727    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.455779 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.455760    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 31 23:58:45.455899 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.455860    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 31 23:58:45.463827 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.463726    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Jul 31 23:58:45.501501 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.501457    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:45.521154 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521125    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls\") pod \"fd22fd29-0892-4089-8b0c-31018be655a2\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") "
Jul 31 23:58:45.521339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521167    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location\") pod \"fd22fd29-0892-4089-8b0c-31018be655a2\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") "
Jul 31 23:58:45.521339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521201    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config\") pod \"fd22fd29-0892-4089-8b0c-31018be655a2\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") "
Jul 31 23:58:45.521339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521260    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert\") pod \"fd22fd29-0892-4089-8b0c-31018be655a2\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") "
Jul 31 23:58:45.521339 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521324    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c76xk\" (UniqueName: \"kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk\") pod \"fd22fd29-0892-4089-8b0c-31018be655a2\" (UID: \"fd22fd29-0892-4089-8b0c-31018be655a2\") "
Jul 31 23:58:45.521578 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521425    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.521578 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521459    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.521578 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521532    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.521714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521614    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd22fd29-0892-4089-8b0c-31018be655a2" (UID: "fd22fd29-0892-4089-8b0c-31018be655a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:45.521714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521627    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rw2\" (UniqueName: \"kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.521714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521645    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd22fd29-0892-4089-8b0c-31018be655a2" (UID: "fd22fd29-0892-4089-8b0c-31018be655a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:45.521714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521684    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd22fd29-0892-4089-8b0c-31018be655a2" (UID: "fd22fd29-0892-4089-8b0c-31018be655a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:45.521714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521702    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.521893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521789    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:45.521893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521805    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd22fd29-0892-4089-8b0c-31018be655a2-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:45.521893 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.521814    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd22fd29-0892-4089-8b0c-31018be655a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:45.523640 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.523596    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk" (OuterVolumeSpecName: "kube-api-access-c76xk") pod "fd22fd29-0892-4089-8b0c-31018be655a2" (UID: "fd22fd29-0892-4089-8b0c-31018be655a2"). InnerVolumeSpecName "kube-api-access-c76xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:58:45.523640 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.523616    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd22fd29-0892-4089-8b0c-31018be655a2" (UID: "fd22fd29-0892-4089-8b0c-31018be655a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:45.622639 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622604    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rw2\" (UniqueName: \"kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622653    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622694    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622713    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622740    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622781    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd22fd29-0892-4089-8b0c-31018be655a2-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:45.622832 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.622796    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76xk\" (UniqueName: \"kubernetes.io/projected/fd22fd29-0892-4089-8b0c-31018be655a2-kube-api-access-c76xk\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:45.623224 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.623205    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.623531 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.623509    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.623580 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.623509    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.625328 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.625303    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.631247 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.631225    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rw2\" (UniqueName: \"kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2\") pod \"isvc-canary-stop-v2-predictor-6688bb854b-7wqgk\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.764962 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.764865    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:45.865456 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.865417    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:45.911269 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:45.911239    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Jul 31 23:58:45.913404 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:58:45.913378    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5902754_bbbe_4554_b3d8_03dc4a2446f6.slice/crio-ab4079cbb6cb6ee35605960de0b208c3e559e709d9921111dbfe006661acec54 WatchSource:0}: Error finding container ab4079cbb6cb6ee35605960de0b208c3e559e709d9921111dbfe006661acec54: Status 404 returned error can't find the container with id ab4079cbb6cb6ee35605960de0b208c3e559e709d9921111dbfe006661acec54
Jul 31 23:58:46.344247 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.344200    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerStarted","Data":"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"}
Jul 31 23:58:46.344736 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.344284    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerStarted","Data":"ab4079cbb6cb6ee35605960de0b208c3e559e709d9921111dbfe006661acec54"}
Jul 31 23:58:46.346405 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.346375    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw" event={"ID":"fd22fd29-0892-4089-8b0c-31018be655a2","Type":"ContainerDied","Data":"639bfac229710a0ec4bb628cbee853bdc8cce8300f5320cfd98ac0c5c304537f"}
Jul 31 23:58:46.346592 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.346405    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"
Jul 31 23:58:46.346592 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.346451    2536 scope.go:117] "RemoveContainer" containerID="612c42436db36c1103fe30f65dad00a76af1fa5563574a4fd674c844c2342d70"
Jul 31 23:58:46.357243 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.357220    2536 scope.go:117] "RemoveContainer" containerID="bbc5c6c98545d0d663c6ec50ba9ec9f6f78e9d849c27dcf4479ad380eef1be92"
Jul 31 23:58:46.367581 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.367560    2536 scope.go:117] "RemoveContainer" containerID="a69fae96a874e10af0e718f7a1e8591d5f792ae2b9595d8f3c6e09fe7c8b4f61"
Jul 31 23:58:46.375907 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.375863    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:58:46.379765 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:46.379735    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-q84jw"]
Jul 31 23:58:47.561728 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:47.561693    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd22fd29-0892-4089-8b0c-31018be655a2" path="/var/lib/kubelet/pods/fd22fd29-0892-4089-8b0c-31018be655a2/volumes"
Jul 31 23:58:50.367770 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.367735    2536 generic.go:356] "Generic (PLEG): container finished" podID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerID="220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844" exitCode=0
Jul 31 23:58:50.368150 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.367780    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerDied","Data":"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"}
Jul 31 23:58:50.431410 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.431369    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.556785 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.556750    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.662128 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.662035    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.699459 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.699422    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:58:50.699754 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.699710    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container" containerID="cri-o://dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769" gracePeriod=30
Jul 31 23:58:50.699962 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.699737    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kube-rbac-proxy" containerID="cri-o://add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78" gracePeriod=30
Jul 31 23:58:50.734610 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.734479    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.791001 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.790968    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.810072 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.810026    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Jul 31 23:58:50.815010 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.814984    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.817319 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.817281    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-64006-kube-rbac-proxy-sar-config\""
Jul 31 23:58:50.817527 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.817498    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-64006-predictor-serving-cert\""
Jul 31 23:58:50.827337 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.827306    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Jul 31 23:58:50.857678 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.857632    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:50.871568 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.871535    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.871765 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.871593    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.871765 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.871706    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsxf\" (UniqueName: \"kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.871861 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.871778    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.871914 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.871899    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972467 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972372    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsxf\" (UniqueName: \"kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972467 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972412    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972467 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972449    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972764 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972530    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972764 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972562    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.972977 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.972957    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.973126 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.973109    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.973226 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.973209    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.975027 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.975005    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:50.980601 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:50.980571    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsxf\" (UniqueName: \"kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf\") pod \"isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:51.128108 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.128065    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:51.278836 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.278802    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Jul 31 23:58:51.281516 ip-10-0-132-129 kubenswrapper[2536]: W0731 23:58:51.281471    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fcaf13_bd5d_4d3d_9340_cb9e8e486a39.slice/crio-b6a4d20de848e8bce931d0cba219d7c08765651d57a8b5ea5f687486105814e0 WatchSource:0}: Error finding container b6a4d20de848e8bce931d0cba219d7c08765651d57a8b5ea5f687486105814e0: Status 404 returned error can't find the container with id b6a4d20de848e8bce931d0cba219d7c08765651d57a8b5ea5f687486105814e0
Jul 31 23:58:51.374635 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.374592    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerStarted","Data":"d0e2a60ff8a4bd899ad629b5995b8dcbd73928c6c4990a9e28c23d9952fb350c"}
Jul 31 23:58:51.375124 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.374689    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerStarted","Data":"b6a4d20de848e8bce931d0cba219d7c08765651d57a8b5ea5f687486105814e0"}
Jul 31 23:58:51.377165 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.377128    2536 generic.go:356] "Generic (PLEG): container finished" podID="319b6709-a459-49b6-9885-a87c4933e97f" containerID="add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78" exitCode=2
Jul 31 23:58:51.377350 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.377201    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerDied","Data":"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"}
Jul 31 23:58:51.380046 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.380012    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerStarted","Data":"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"}
Jul 31 23:58:51.380203 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.380088    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerStarted","Data":"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"}
Jul 31 23:58:51.380343 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.380321    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:51.408714 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:51.408662    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podStartSLOduration=2.661162369 podStartE2EDuration="6.408641003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747478634s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:58:51.406972143 +0000 UTC m=+756.372167790" watchObservedRunningTime="2026-07-31 23:58:51.408641003 +0000 UTC m=+756.373836626"
Jul 31 23:58:52.386114 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:52.386073    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:52.387400 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:52.387371    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:58:53.390624 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:53.390578    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:58:54.863507 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.863458    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:58:54.906785 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.906753    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls\") pod \"319b6709-a459-49b6-9885-a87c4933e97f\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") "
Jul 31 23:58:54.906941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.906843    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config\") pod \"319b6709-a459-49b6-9885-a87c4933e97f\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") "
Jul 31 23:58:54.906941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.906865    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdmkj\" (UniqueName: \"kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj\") pod \"319b6709-a459-49b6-9885-a87c4933e97f\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") "
Jul 31 23:58:54.906941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.906904    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert\") pod \"319b6709-a459-49b6-9885-a87c4933e97f\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") "
Jul 31 23:58:54.906941 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.906938    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location\") pod \"319b6709-a459-49b6-9885-a87c4933e97f\" (UID: \"319b6709-a459-49b6-9885-a87c4933e97f\") "
Jul 31 23:58:54.907300 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.907268    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "319b6709-a459-49b6-9885-a87c4933e97f" (UID: "319b6709-a459-49b6-9885-a87c4933e97f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:54.907425 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.907284    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "319b6709-a459-49b6-9885-a87c4933e97f" (UID: "319b6709-a459-49b6-9885-a87c4933e97f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:54.907566 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.907421    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "319b6709-a459-49b6-9885-a87c4933e97f" (UID: "319b6709-a459-49b6-9885-a87c4933e97f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:54.909073 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.909043    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj" (OuterVolumeSpecName: "kube-api-access-hdmkj") pod "319b6709-a459-49b6-9885-a87c4933e97f" (UID: "319b6709-a459-49b6-9885-a87c4933e97f"). InnerVolumeSpecName "kube-api-access-hdmkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:58:54.909156 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:54.909099    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "319b6709-a459-49b6-9885-a87c4933e97f" (UID: "319b6709-a459-49b6-9885-a87c4933e97f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:55.007947 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.007912    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/319b6709-a459-49b6-9885-a87c4933e97f-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:55.007947 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.007945    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:55.008330 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.007961    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmkj\" (UniqueName: \"kubernetes.io/projected/319b6709-a459-49b6-9885-a87c4933e97f-kube-api-access-hdmkj\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:55.008330 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.007970    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/319b6709-a459-49b6-9885-a87c4933e97f-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:55.008330 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.007979    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/319b6709-a459-49b6-9885-a87c4933e97f-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Jul 31 23:58:55.238014 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.237914    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:55.401956 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.401918    2536 generic.go:356] "Generic (PLEG): container finished" podID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerID="d0e2a60ff8a4bd899ad629b5995b8dcbd73928c6c4990a9e28c23d9952fb350c" exitCode=0
Jul 31 23:58:55.402163 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.402005    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerDied","Data":"d0e2a60ff8a4bd899ad629b5995b8dcbd73928c6c4990a9e28c23d9952fb350c"}
Jul 31 23:58:55.404358 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.404326    2536 generic.go:356] "Generic (PLEG): container finished" podID="319b6709-a459-49b6-9885-a87c4933e97f" containerID="dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769" exitCode=0
Jul 31 23:58:55.404522 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.404394    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"
Jul 31 23:58:55.404522 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.404446    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerDied","Data":"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"}
Jul 31 23:58:55.404644 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.404532    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl" event={"ID":"319b6709-a459-49b6-9885-a87c4933e97f","Type":"ContainerDied","Data":"978e28b83bc19b6c6f2bc6e281e58d123648248568b9c8203a557bbad6c9eca3"}
Jul 31 23:58:55.404644 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.404577    2536 scope.go:117] "RemoveContainer" containerID="add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"
Jul 31 23:58:55.417355 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.417330    2536 scope.go:117] "RemoveContainer" containerID="dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"
Jul 31 23:58:55.426704 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.426675    2536 scope.go:117] "RemoveContainer" containerID="6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"
Jul 31 23:58:55.435913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.435881    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:58:55.443453 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.443425    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-4da04-predictor-6f684fdd5c-g7zcl"]
Jul 31 23:58:55.446636 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.446616    2536 scope.go:117] "RemoveContainer" containerID="add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"
Jul 31 23:58:55.447027 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:58:55.447004    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78\": container with ID starting with add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78 not found: ID does not exist" containerID="add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"
Jul 31 23:58:55.447113 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.447041    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78"} err="failed to get container status \"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78\": rpc error: code = NotFound desc = could not find container \"add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78\": container with ID starting with add504d5a2331e8e16f3f9e7c612dba489e80e19e67595e1071f7ebc90faeb78 not found: ID does not exist"
Jul 31 23:58:55.447113 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.447069    2536 scope.go:117] "RemoveContainer" containerID="dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"
Jul 31 23:58:55.447379 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:58:55.447359    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769\": container with ID starting with dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769 not found: ID does not exist" containerID="dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"
Jul 31 23:58:55.447445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.447386    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769"} err="failed to get container status \"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769\": rpc error: code = NotFound desc = could not find container \"dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769\": container with ID starting with dfd9cd17fed56009fb12b61f37a0da30400477c474610e6b3693e58a217fd769 not found: ID does not exist"
Jul 31 23:58:55.447445 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.447409    2536 scope.go:117] "RemoveContainer" containerID="6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"
Jul 31 23:58:55.447708 ip-10-0-132-129 kubenswrapper[2536]: E0731 23:58:55.447686    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c\": container with ID starting with 6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c not found: ID does not exist" containerID="6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"
Jul 31 23:58:55.447776 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.447713    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c"} err="failed to get container status \"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c\": rpc error: code = NotFound desc = could not find container \"6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c\": container with ID starting with 6abd83af584af5d2db318a3fcb93d2e8e1cf9090150fd9c375ef9c17ad34266c not found: ID does not exist"
Jul 31 23:58:55.568587 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:55.568543    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319b6709-a459-49b6-9885-a87c4933e97f" path="/var/lib/kubelet/pods/319b6709-a459-49b6-9885-a87c4933e97f/volumes"
Jul 31 23:58:56.413164 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:56.413125    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerStarted","Data":"63f0ce762496e3a0d000335ca2d597e1d8775ae3b672c1f8964a502919a9942e"}
Jul 31 23:58:56.413673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:56.413194    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerStarted","Data":"d86a93b2cf3990a48ce570268af1a4ef717ac5ad5618c1ede9fcaedc499bd66c"}
Jul 31 23:58:56.413673 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:56.413418    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:56.433206 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:56.433145    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podStartSLOduration=2.803384113 podStartE2EDuration="6.433127205s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629743092s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:58:56.431554184 +0000 UTC m=+761.396749809" watchObservedRunningTime="2026-07-31 23:58:56.433127205 +0000 UTC m=+761.398322897"
Jul 31 23:58:57.420295 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:57.420261    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:58:57.421443 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:57.421411    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:58:58.395962 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:58.395930    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Jul 31 23:58:58.396576 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:58.396544    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:58:58.424789 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:58:58.424747    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:03.429982 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:03.429946    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Jul 31 23:59:03.430637 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:03.430606    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:08.396858 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:08.396808    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:59:13.431198 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:13.431159    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:18.396411 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:18.396364    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:59:23.431034 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:23.430917    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:28.397328 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:28.397275    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:59:33.431447 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:33.431405    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:38.397006 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:38.396915    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:59:43.431201 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:43.431152    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:48.396869 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:48.396825    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 31 23:59:53.430913 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:53.430873    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 31 23:59:58.397194 ip-10-0-132-129 kubenswrapper[2536]: I0731 23:59:58.397147    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 01 00:00:03.431679 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:03.431643    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Aug 01 00:00:08.397170 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:08.397135    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Aug 01 00:00:15.431251 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.431212    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:15.447741 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.447701    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Aug 01 00:00:15.447968 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.447942    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" containerID="cri-o://bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d" gracePeriod=30
Aug 01 00:00:15.448039 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.447975    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kube-rbac-proxy" containerID="cri-o://86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751" gracePeriod=30
Aug 01 00:00:15.529112 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.529063    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:15.805637 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.805597    2536 generic.go:356] "Generic (PLEG): container finished" podID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerID="86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751" exitCode=2
Aug 01 00:00:15.805828 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:15.805676    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerDied","Data":"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"}
Aug 01 00:00:18.391704 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:18.391651    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 01 00:00:18.397334 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:18.397300    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 01 00:00:20.815317 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.815290    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Aug 01 00:00:20.834128 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.834085    2536 generic.go:356] "Generic (PLEG): container finished" podID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerID="bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d" exitCode=0
Aug 01 00:00:20.834562 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.834167    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerDied","Data":"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"}
Aug 01 00:00:20.834562 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.834204    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"
Aug 01 00:00:20.834562 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.834246    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk" event={"ID":"f5902754-bbbe-4554-b3d8-03dc4a2446f6","Type":"ContainerDied","Data":"ab4079cbb6cb6ee35605960de0b208c3e559e709d9921111dbfe006661acec54"}
Aug 01 00:00:20.834562 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.834283    2536 scope.go:117] "RemoveContainer" containerID="86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"
Aug 01 00:00:20.844627 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.844604    2536 scope.go:117] "RemoveContainer" containerID="bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"
Aug 01 00:00:20.853460 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.853424    2536 scope.go:117] "RemoveContainer" containerID="220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"
Aug 01 00:00:20.863520 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.863459    2536 scope.go:117] "RemoveContainer" containerID="86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"
Aug 01 00:00:20.863896 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:00:20.863858    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751\": container with ID starting with 86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751 not found: ID does not exist" containerID="86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"
Aug 01 00:00:20.864034 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.863900    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751"} err="failed to get container status \"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751\": rpc error: code = NotFound desc = could not find container \"86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751\": container with ID starting with 86af7215b486c0c703b4d28eb204a4e226aa00f16fd0c5258053a5441d470751 not found: ID does not exist"
Aug 01 00:00:20.864034 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.863926    2536 scope.go:117] "RemoveContainer" containerID="bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"
Aug 01 00:00:20.864284 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:00:20.864263    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d\": container with ID starting with bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d not found: ID does not exist" containerID="bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"
Aug 01 00:00:20.864352 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.864294    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d"} err="failed to get container status \"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d\": rpc error: code = NotFound desc = could not find container \"bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d\": container with ID starting with bc9882ab65b2f08282437c208eab2f45b0095e2bcbc6740948ec251a07f3910d not found: ID does not exist"
Aug 01 00:00:20.864352 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.864311    2536 scope.go:117] "RemoveContainer" containerID="220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"
Aug 01 00:00:20.864677 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:00:20.864660    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844\": container with ID starting with 220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844 not found: ID does not exist" containerID="220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"
Aug 01 00:00:20.864747 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.864680    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844"} err="failed to get container status \"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844\": rpc error: code = NotFound desc = could not find container \"220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844\": container with ID starting with 220864594ce1bf4f6c656d7247edd7dd0807d3bfd4e98d23ccd8728aa103c844 not found: ID does not exist"
Aug 01 00:00:20.924941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.924897    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert\") pod \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") "
Aug 01 00:00:20.924941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.924945    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location\") pod \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") "
Aug 01 00:00:20.925166 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.924989    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5rw2\" (UniqueName: \"kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2\") pod \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") "
Aug 01 00:00:20.925166 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925087    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls\") pod \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") "
Aug 01 00:00:20.925253 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925178    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config\") pod \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\" (UID: \"f5902754-bbbe-4554-b3d8-03dc4a2446f6\") "
Aug 01 00:00:20.925338 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925309    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5902754-bbbe-4554-b3d8-03dc4a2446f6" (UID: "f5902754-bbbe-4554-b3d8-03dc4a2446f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:20.925338 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925329    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5902754-bbbe-4554-b3d8-03dc4a2446f6" (UID: "f5902754-bbbe-4554-b3d8-03dc4a2446f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:20.925540 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925520    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.925540 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925540    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.925706 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.925604    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5902754-bbbe-4554-b3d8-03dc4a2446f6" (UID: "f5902754-bbbe-4554-b3d8-03dc4a2446f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:20.927185 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.927164    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5902754-bbbe-4554-b3d8-03dc4a2446f6" (UID: "f5902754-bbbe-4554-b3d8-03dc4a2446f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:20.927305 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:20.927281    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2" (OuterVolumeSpecName: "kube-api-access-c5rw2") pod "f5902754-bbbe-4554-b3d8-03dc4a2446f6" (UID: "f5902754-bbbe-4554-b3d8-03dc4a2446f6"). InnerVolumeSpecName "kube-api-access-c5rw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:21.026984 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.026943    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:21.026984 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.026977    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rw2\" (UniqueName: \"kubernetes.io/projected/f5902754-bbbe-4554-b3d8-03dc4a2446f6-kube-api-access-c5rw2\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:21.026984 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.026992    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5902754-bbbe-4554-b3d8-03dc4a2446f6-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:21.156033 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.155992    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Aug 01 00:00:21.159913 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.159880    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6688bb854b-7wqgk"]
Aug 01 00:00:21.567182 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:21.567146    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" path="/var/lib/kubelet/pods/f5902754-bbbe-4554-b3d8-03dc4a2446f6/volumes"
Aug 01 00:00:25.647870 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:25.647813    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:26.075179 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:26.075135    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:30.656590 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:30.656550    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:30.873308 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:30.873266    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:40.733244 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.733196    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:40.852561 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.852525    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:40.876917 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.876880    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:00:40.877306 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877291    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877307    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877317    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="storage-initializer"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877323    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="storage-initializer"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877338    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="storage-initializer"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877344    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="storage-initializer"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877358    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877366    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container"
Aug 01 00:00:40.877377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877379    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877384    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877397    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877402    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877466    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877477    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kube-rbac-proxy"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877499    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="319b6709-a459-49b6-9885-a87c4933e97f" containerName="kserve-container"
Aug 01 00:00:40.877806 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.877505    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5902754-bbbe-4554-b3d8-03dc4a2446f6" containerName="kserve-container"
Aug 01 00:00:40.882990 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.882959    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:40.885772 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.885731    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-40d2c-kube-rbac-proxy-sar-config\""
Aug 01 00:00:40.885927 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.885796    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-40d2c-predictor-serving-cert\""
Aug 01 00:00:40.887709 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.887685    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:00:40.892356 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.892328    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Aug 01 00:00:40.892690 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.892637    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" containerID="cri-o://d86a93b2cf3990a48ce570268af1a4ef717ac5ad5618c1ede9fcaedc499bd66c" gracePeriod=30
Aug 01 00:00:40.893081 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.892683    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kube-rbac-proxy" containerID="cri-o://63f0ce762496e3a0d000335ca2d597e1d8775ae3b672c1f8964a502919a9942e" gracePeriod=30
Aug 01 00:00:40.908362 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.908331    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:40.908527 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.908382    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:40.908527 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.908438    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5g8\" (UniqueName: \"kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:40.925846 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.925812    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:40.999641 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:40.999546    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:41.009098 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.009070    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.009280 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.009118    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.009280 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.009219    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5g8\" (UniqueName: \"kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.009838 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.009797    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.011979 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.011946    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.016569 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.016538    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5g8\" (UniqueName: \"kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8\") pod \"message-dumper-raw-40d2c-predictor-866c4c55-pvft6\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") " pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.198799 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.198750    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:41.342452 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.342276    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:00:41.345354 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:00:41.345323    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3811d60_e94d_4fec_aa84_5df7f478d881.slice/crio-141ff0e0669188c16f93470798df539ac3697c0a5c34a589bb587a596f72ef17 WatchSource:0}: Error finding container 141ff0e0669188c16f93470798df539ac3697c0a5c34a589bb587a596f72ef17: Status 404 returned error can't find the container with id 141ff0e0669188c16f93470798df539ac3697c0a5c34a589bb587a596f72ef17
Aug 01 00:00:41.347310 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.347294    2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:00:41.940548 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.940509    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerStarted","Data":"141ff0e0669188c16f93470798df539ac3697c0a5c34a589bb587a596f72ef17"}
Aug 01 00:00:41.942750 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.942721    2536 generic.go:356] "Generic (PLEG): container finished" podID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerID="63f0ce762496e3a0d000335ca2d597e1d8775ae3b672c1f8964a502919a9942e" exitCode=2
Aug 01 00:00:41.942904 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:41.942795    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerDied","Data":"63f0ce762496e3a0d000335ca2d597e1d8775ae3b672c1f8964a502919a9942e"}
Aug 01 00:00:42.950105 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:42.950058    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerStarted","Data":"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"}
Aug 01 00:00:42.950524 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:42.950127    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerStarted","Data":"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"}
Aug 01 00:00:42.950524 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:42.950254    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:43.425290 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:43.425243    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Aug 01 00:00:43.430922 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:43.430882    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 01 00:00:43.955152 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:43.955102    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:43.957311 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:43.957282    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:00:43.973702 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:43.973645    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" podStartSLOduration=2.813945064 podStartE2EDuration="3.973627906s" totalImagesPullingTime="1.159682842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:42.970734852 +0000 UTC m=+867.935930476" watchObservedRunningTime="2026-08-01 00:00:43.973627906 +0000 UTC m=+868.938823540"
Aug 01 00:00:44.962752 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:44.962714    2536 generic.go:356] "Generic (PLEG): container finished" podID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerID="d86a93b2cf3990a48ce570268af1a4ef717ac5ad5618c1ede9fcaedc499bd66c" exitCode=0
Aug 01 00:00:44.963121 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:44.962784    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerDied","Data":"d86a93b2cf3990a48ce570268af1a4ef717ac5ad5618c1ede9fcaedc499bd66c"}
Aug 01 00:00:45.054842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.054772    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Aug 01 00:00:45.149734 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.149694    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location\") pod \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") "
Aug 01 00:00:45.149937 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.149743    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbsxf\" (UniqueName: \"kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf\") pod \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") "
Aug 01 00:00:45.149937 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.149766    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert\") pod \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") "
Aug 01 00:00:45.149937 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.149819    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls\") pod \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") "
Aug 01 00:00:45.149937 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.149879    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config\") pod \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\" (UID: \"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39\") "
Aug 01 00:00:45.150155 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.150078    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" (UID: "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:45.150221 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.150173    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:45.150275 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.150237    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" (UID: "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:45.150382 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.150358    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" (UID: "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:45.152011 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.151980    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" (UID: "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:45.152011 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.151989    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf" (OuterVolumeSpecName: "kube-api-access-fbsxf") pod "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" (UID: "17fcaf13-bd5d-4d3d-9340-cb9e8e486a39"). InnerVolumeSpecName "kube-api-access-fbsxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:45.251445 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.251402    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbsxf\" (UniqueName: \"kubernetes.io/projected/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-api-access-fbsxf\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:45.251445 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.251440    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:45.251445 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.251450    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:45.251718 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.251460    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:00:45.970553 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.970463    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"
Aug 01 00:00:45.970553 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.970502    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc" event={"ID":"17fcaf13-bd5d-4d3d-9340-cb9e8e486a39","Type":"ContainerDied","Data":"b6a4d20de848e8bce931d0cba219d7c08765651d57a8b5ea5f687486105814e0"}
Aug 01 00:00:45.971051 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.970572    2536 scope.go:117] "RemoveContainer" containerID="63f0ce762496e3a0d000335ca2d597e1d8775ae3b672c1f8964a502919a9942e"
Aug 01 00:00:45.979924 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.979899    2536 scope.go:117] "RemoveContainer" containerID="d86a93b2cf3990a48ce570268af1a4ef717ac5ad5618c1ede9fcaedc499bd66c"
Aug 01 00:00:45.988378 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.988343    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Aug 01 00:00:45.989572 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.989546    2536 scope.go:117] "RemoveContainer" containerID="d0e2a60ff8a4bd899ad629b5995b8dcbd73928c6c4990a9e28c23d9952fb350c"
Aug 01 00:00:45.991687 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:45.991664    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-64006-predictor-648db47f7c-txxnc"]
Aug 01 00:00:47.567357 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:47.567320    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" path="/var/lib/kubelet/pods/17fcaf13-bd5d-4d3d-9340-cb9e8e486a39/volumes"
Aug 01 00:00:50.976184 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:00:50.976155    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:01:01.386247 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:01:01.386210    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:01.487003 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:01:01.486967    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:15.536048 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:01:15.536018    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:01:15.537909 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:01:15.537876    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:02:10.835069 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.835017    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:10.863438 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863392    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:10.863931 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863911    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="storage-initializer"
Aug 01 00:02:10.863931 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863932    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="storage-initializer"
Aug 01 00:02:10.864095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863949    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container"
Aug 01 00:02:10.864095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863957    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container"
Aug 01 00:02:10.864095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863989    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kube-rbac-proxy"
Aug 01 00:02:10.864095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.863998    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kube-rbac-proxy"
Aug 01 00:02:10.864300 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.864102    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kserve-container"
Aug 01 00:02:10.864300 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.864116    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fcaf13-bd5d-4d3d-9340-cb9e8e486a39" containerName="kube-rbac-proxy"
Aug 01 00:02:10.867723 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.867693    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:10.870201 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.870162    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-53f567-kube-rbac-proxy-sar-config\""
Aug 01 00:02:10.870201 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.870168    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-53f567-dockercfg-jgdv8\""
Aug 01 00:02:10.870430 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.870215    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-53f567-predictor-serving-cert\""
Aug 01 00:02:10.870430 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.870202    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-53f567\""
Aug 01 00:02:10.870430 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.870330    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:02:10.877280 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.877225    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:10.904219 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.904178    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:10.904511 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.904225    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlfm\" (UniqueName: \"kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:10.904511 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.904350    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:10.904511 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.904452    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:10.904511 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:10.904481    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005050 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005005    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005050 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005057    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlfm\" (UniqueName: \"kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005336 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005106    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005336 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005179    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005336 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005204    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005336 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:11.005300    2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-53f567-predictor-serving-cert: secret "isvc-secondary-53f567-predictor-serving-cert" not found
Aug 01 00:02:11.005585 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:11.005383    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls podName:a82c2850-da09-4e15-9d4d-1bbd43a9e3b1 nodeName:}" failed. No retries permitted until 2026-08-01 00:02:11.505360948 +0000 UTC m=+956.470556565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls") pod "isvc-secondary-53f567-predictor-79d64584fb-z58fk" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1") : secret "isvc-secondary-53f567-predictor-serving-cert" not found
Aug 01 00:02:11.005585 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005545    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005845 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005820    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.005969 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.005949    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.015930 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.015897    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlfm\" (UniqueName: \"kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.509527 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.509464    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.511952 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.511928    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") pod \"isvc-secondary-53f567-predictor-79d64584fb-z58fk\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") " pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.780731 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.780696    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:11.906436 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.906399    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:11.912557 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:11.912529    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:11.914264 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:02:11.914236    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82c2850_da09_4e15_9d4d_1bbd43a9e3b1.slice/crio-658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b WatchSource:0}: Error finding container 658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b: Status 404 returned error can't find the container with id 658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b
Aug 01 00:02:12.130883 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:12.130778    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:12.384700 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:12.384601    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerStarted","Data":"ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"}
Aug 01 00:02:12.384700 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:12.384662    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerStarted","Data":"658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b"}
Aug 01 00:02:17.410223 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:17.410192    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/0.log"
Aug 01 00:02:17.410694 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:17.410235    2536 generic.go:356] "Generic (PLEG): container finished" podID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerID="ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01" exitCode=1
Aug 01 00:02:17.410694 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:17.410317    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerDied","Data":"ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"}
Aug 01 00:02:18.416994 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:18.416962    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/0.log"
Aug 01 00:02:18.417429 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:18.417076    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerStarted","Data":"bbe0e7fa350e5a01a201d90b1451f29e8c4d8c616687e07d39c55f733fa15bb3"}
Aug 01 00:02:22.437951 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.437921    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/1.log"
Aug 01 00:02:22.438356 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.438332    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/0.log"
Aug 01 00:02:22.438413 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.438369    2536 generic.go:356] "Generic (PLEG): container finished" podID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerID="bbe0e7fa350e5a01a201d90b1451f29e8c4d8c616687e07d39c55f733fa15bb3" exitCode=1
Aug 01 00:02:22.438464 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.438428    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerDied","Data":"bbe0e7fa350e5a01a201d90b1451f29e8c4d8c616687e07d39c55f733fa15bb3"}
Aug 01 00:02:22.438538 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.438523    2536 scope.go:117] "RemoveContainer" containerID="ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"
Aug 01 00:02:22.438951 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.438930    2536 scope.go:117] "RemoveContainer" containerID="ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"
Aug 01 00:02:22.450361 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:22.450324    2536 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-53f567-predictor-79d64584fb-z58fk_kserve-ci-e2e-test(a82c2850-da09-4e15-9d4d-1bbd43a9e3b1)\"" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"
Aug 01 00:02:22.450469 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:22.450341    2536 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-53f567-predictor-79d64584fb-z58fk_kserve-ci-e2e-test_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1_0 in pod sandbox 658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b from index: no such id: 'ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01'" containerID="ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"
Aug 01 00:02:22.450469 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:22.450402    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-53f567-predictor-79d64584fb-z58fk_kserve-ci-e2e-test_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1_0 in pod sandbox 658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b from index: no such id: 'ce0a97c7e7bf7bc76448f599e280dcad1370e3373f6ccd97861166e085ae2a01'"
Aug 01 00:02:23.444863 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:23.444829    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/1.log"
Aug 01 00:02:28.901710 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:28.901665    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:28.930247 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:28.930200    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:29.030417 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.030340    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:29.053671 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.053646    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/1.log"
Aug 01 00:02:29.057283 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.057257    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:29.168330 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168237    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config\") pod \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") "
Aug 01 00:02:29.168330 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168326    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") pod \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") "
Aug 01 00:02:29.168625 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168353    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlfm\" (UniqueName: \"kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm\") pod \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") "
Aug 01 00:02:29.168625 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168405    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location\") pod \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") "
Aug 01 00:02:29.168625 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168461    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert\") pod \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\" (UID: \"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1\") "
Aug 01 00:02:29.168787 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168636    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:02:29.168787 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168745    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:29.168787 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168775    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:29.168969 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.168942    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:02:29.170530 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.170501    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm" (OuterVolumeSpecName: "kube-api-access-5tlfm") pod "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"). InnerVolumeSpecName "kube-api-access-5tlfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:02:29.170607 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.170524    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" (UID: "a82c2850-da09-4e15-9d4d-1bbd43a9e3b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:02:29.270354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.270298    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:29.270354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.270344    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:29.270354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.270354    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlfm\" (UniqueName: \"kubernetes.io/projected/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kube-api-access-5tlfm\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:29.270354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.270364    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:29.475716 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.475634    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-53f567-predictor-79d64584fb-z58fk_a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/storage-initializer/1.log"
Aug 01 00:02:29.475716 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.475709    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk" event={"ID":"a82c2850-da09-4e15-9d4d-1bbd43a9e3b1","Type":"ContainerDied","Data":"658ff9879b30055625a93bd7e6652d9d52b551ee601d9b02c582bf0fd289ad7b"}
Aug 01 00:02:29.475899 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.475763    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"
Aug 01 00:02:29.475899 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.475791    2536 scope.go:117] "RemoveContainer" containerID="bbe0e7fa350e5a01a201d90b1451f29e8c4d8c616687e07d39c55f733fa15bb3"
Aug 01 00:02:29.514367 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.514120    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:29.517570 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.517543    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-53f567-predictor-79d64584fb-z58fk"]
Aug 01 00:02:29.561575 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:29.561540    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" path="/var/lib/kubelet/pods/a82c2850-da09-4e15-9d4d-1bbd43a9e3b1/volumes"
Aug 01 00:02:34.122152 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:34.122112    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:34.633509 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:34.633449    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:35.956667 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:35.956631    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-40d2c-predictor-866c4c55-pvft6_b3811d60-e94d-4fec-aa84-5df7f478d881/kserve-container/0.log"
Aug 01 00:02:36.155275 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.155233    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:36.181768 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.181730    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:02:36.182039 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.181987    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kserve-container" containerID="cri-o://8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227" gracePeriod=30
Aug 01 00:02:36.182168 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.182025    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kube-rbac-proxy" containerID="cri-o://c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71" gracePeriod=30
Aug 01 00:02:36.210346 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210261    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:02:36.210693 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210679    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.210756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210695    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.210756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210711    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.210756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210716    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.210848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210785    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.210917 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.210908    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82c2850-da09-4e15-9d4d-1bbd43a9e3b1" containerName="storage-initializer"
Aug 01 00:02:36.214253 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.214224    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.216797 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.216756    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-83d7e-kube-rbac-proxy-sar-config\""
Aug 01 00:02:36.216971 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.216756    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-83d7e-predictor-serving-cert\""
Aug 01 00:02:36.216971 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.216759    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:02:36.217432 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.217415    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:02:36.223308 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.223275    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:02:36.241938 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.241895    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:36.342150 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.342115    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.342150 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.342154    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.342360 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.342173    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.342360 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.342258    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbl4p\" (UniqueName: \"kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.342435 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.342367    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.442818 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.442778    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbl4p\" (UniqueName: \"kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.442998 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.442863    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.442998 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.442889    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.442998 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.442909    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.442998 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.442930    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.443206 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:36.443034    2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-serving-cert: secret "isvc-sklearn-scale-raw-83d7e-predictor-serving-cert" not found
Aug 01 00:02:36.443206 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:36.443119    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls podName:a2834430-a979-475c-993d-b9ddc36b85b0 nodeName:}" failed. No retries permitted until 2026-08-01 00:02:36.943096985 +0000 UTC m=+981.908292585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls") pod "isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0") : secret "isvc-sklearn-scale-raw-83d7e-predictor-serving-cert" not found
Aug 01 00:02:36.443366 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.443345    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.443634 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.443613    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.443740 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.443648    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.445954 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.445932    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:02:36.453173 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.453142    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbl4p\" (UniqueName: \"kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.511842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511733    2536 generic.go:356] "Generic (PLEG): container finished" podID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerID="c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71" exitCode=2
Aug 01 00:02:36.511842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511763    2536 generic.go:356] "Generic (PLEG): container finished" podID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerID="8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227" exitCode=2
Aug 01 00:02:36.511842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511787    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"
Aug 01 00:02:36.511842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511796    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerDied","Data":"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"}
Aug 01 00:02:36.512168 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511858    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerDied","Data":"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"}
Aug 01 00:02:36.512168 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511884    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6" event={"ID":"b3811d60-e94d-4fec-aa84-5df7f478d881","Type":"ContainerDied","Data":"141ff0e0669188c16f93470798df539ac3697c0a5c34a589bb587a596f72ef17"}
Aug 01 00:02:36.512168 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.511912    2536 scope.go:117] "RemoveContainer" containerID="c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"
Aug 01 00:02:36.520654 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.520633    2536 scope.go:117] "RemoveContainer" containerID="8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"
Aug 01 00:02:36.531452 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.531427    2536 scope.go:117] "RemoveContainer" containerID="c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"
Aug 01 00:02:36.531824 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:36.531804    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71\": container with ID starting with c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71 not found: ID does not exist" containerID="c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"
Aug 01 00:02:36.531888 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.531834    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"} err="failed to get container status \"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71\": rpc error: code = NotFound desc = could not find container \"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71\": container with ID starting with c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71 not found: ID does not exist"
Aug 01 00:02:36.531888 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.531852    2536 scope.go:117] "RemoveContainer" containerID="8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"
Aug 01 00:02:36.532072 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:02:36.532056    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227\": container with ID starting with 8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227 not found: ID does not exist" containerID="8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"
Aug 01 00:02:36.532110 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.532076    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"} err="failed to get container status \"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227\": rpc error: code = NotFound desc = could not find container \"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227\": container with ID starting with 8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227 not found: ID does not exist"
Aug 01 00:02:36.532110 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.532089    2536 scope.go:117] "RemoveContainer" containerID="c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"
Aug 01 00:02:36.532344 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.532314    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71"} err="failed to get container status \"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71\": rpc error: code = NotFound desc = could not find container \"c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71\": container with ID starting with c15179dd63ea97040dc152746b48807f7e43f5b96085a2136efe262fbf63fc71 not found: ID does not exist"
Aug 01 00:02:36.532344 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.532341    2536 scope.go:117] "RemoveContainer" containerID="8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"
Aug 01 00:02:36.532888 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.532868    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227"} err="failed to get container status \"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227\": rpc error: code = NotFound desc = could not find container \"8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227\": container with ID starting with 8fd7238806c92f35c16fbbc424b895dc342bf1e28171903afd565caf78bc9227 not found: ID does not exist"
Aug 01 00:02:36.543998 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.543965    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n5g8\" (UniqueName: \"kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8\") pod \"b3811d60-e94d-4fec-aa84-5df7f478d881\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") "
Aug 01 00:02:36.544175 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.544163    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config\") pod \"b3811d60-e94d-4fec-aa84-5df7f478d881\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") "
Aug 01 00:02:36.544235 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.544192    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls\") pod \"b3811d60-e94d-4fec-aa84-5df7f478d881\" (UID: \"b3811d60-e94d-4fec-aa84-5df7f478d881\") "
Aug 01 00:02:36.544561 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.544531    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3811d60-e94d-4fec-aa84-5df7f478d881" (UID: "b3811d60-e94d-4fec-aa84-5df7f478d881"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:02:36.546144 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.546116    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8" (OuterVolumeSpecName: "kube-api-access-2n5g8") pod "b3811d60-e94d-4fec-aa84-5df7f478d881" (UID: "b3811d60-e94d-4fec-aa84-5df7f478d881"). InnerVolumeSpecName "kube-api-access-2n5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:02:36.546237 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.546174    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3811d60-e94d-4fec-aa84-5df7f478d881" (UID: "b3811d60-e94d-4fec-aa84-5df7f478d881"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:02:36.620850 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.620813    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:36.645620 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.645579    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:36.645620 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.645611    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3811d60-e94d-4fec-aa84-5df7f478d881-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:36.645620 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.645621    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5g8\" (UniqueName: \"kubernetes.io/projected/b3811d60-e94d-4fec-aa84-5df7f478d881-kube-api-access-2n5g8\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:02:36.835627 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.835594    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:02:36.838313 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.838282    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-40d2c-predictor-866c4c55-pvft6"]
Aug 01 00:02:36.949158 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.949115    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:36.951609 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:36.951586    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") pod \"isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:37.169184 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:37.169094    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:37.310185 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:37.309947    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:02:37.312810 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:02:37.312778    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2834430_a979_475c_993d_b9ddc36b85b0.slice/crio-612632f37fdd2137157022b1f1117616d167133693ce818d3e79d2361e3687d7 WatchSource:0}: Error finding container 612632f37fdd2137157022b1f1117616d167133693ce818d3e79d2361e3687d7: Status 404 returned error can't find the container with id 612632f37fdd2137157022b1f1117616d167133693ce818d3e79d2361e3687d7
Aug 01 00:02:37.519543 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:37.519434    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerStarted","Data":"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"}
Aug 01 00:02:37.519543 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:37.519526    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerStarted","Data":"612632f37fdd2137157022b1f1117616d167133693ce818d3e79d2361e3687d7"}
Aug 01 00:02:37.562500 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:37.562440    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" path="/var/lib/kubelet/pods/b3811d60-e94d-4fec-aa84-5df7f478d881/volumes"
Aug 01 00:02:39.154989 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:39.154953    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:41.540928 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:41.540895    2536 generic.go:356] "Generic (PLEG): container finished" podID="a2834430-a979-475c-993d-b9ddc36b85b0" containerID="2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40" exitCode=0
Aug 01 00:02:41.541331 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:41.540948    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerDied","Data":"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"}
Aug 01 00:02:42.548230 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:42.548194    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerStarted","Data":"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"}
Aug 01 00:02:42.548766 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:42.548261    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerStarted","Data":"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"}
Aug 01 00:02:42.548766 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:42.548585    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:42.567047 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:42.566995    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podStartSLOduration=2.728051439 podStartE2EDuration="6.566978948s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838927509s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:02:42.566151938 +0000 UTC m=+987.531347565" watchObservedRunningTime="2026-08-01 00:02:42.566978948 +0000 UTC m=+987.532174571"
Aug 01 00:02:43.553189 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:43.553154    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:43.554665 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:43.554628    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:02:44.316866 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.316827    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:44.349683 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.349644    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:02:44.350141 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350116    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kserve-container"
Aug 01 00:02:44.350141 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350143    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kserve-container"
Aug 01 00:02:44.350307 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350169    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kube-rbac-proxy"
Aug 01 00:02:44.350307 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350174    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kube-rbac-proxy"
Aug 01 00:02:44.350307 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350259    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kube-rbac-proxy"
Aug 01 00:02:44.350307 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.350267    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3811d60-e94d-4fec-aa84-5df7f478d881" containerName="kserve-container"
Aug 01 00:02:44.353676 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.353653    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.355892 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.355868    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-7288f-kube-rbac-proxy-sar-config\""
Aug 01 00:02:44.356022 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.355869    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-7288f-predictor-serving-cert\""
Aug 01 00:02:44.363293 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.363262    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:02:44.521871 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.521835    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.521871 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.521878    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.522149 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.521903    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.522149 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.521980    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2q7s\" (UniqueName: \"kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.522149 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.522029    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.557650 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.557614    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:02:44.623038 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.622958    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623038 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623005    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623238 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623127    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623238 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623165    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2q7s\" (UniqueName: \"kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623287    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623415 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623398    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623679 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623657    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.623851 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.623830    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.625753 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.625733    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.631610 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.631588    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2q7s\" (UniqueName: \"kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s\") pod \"raw-sklearn-7288f-predictor-799b9b69d-r56bl\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") " pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.667620 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.667580    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:44.672775 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.672745    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:44.807262 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:44.807231    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:02:44.809650 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:02:44.809622    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f2cb58_9568_4981_811f_2ce5fdebe58f.slice/crio-db7b83dd409ada0309ee08322f020042147c108263047e9353da45c7e0030bab WatchSource:0}: Error finding container db7b83dd409ada0309ee08322f020042147c108263047e9353da45c7e0030bab: Status 404 returned error can't find the container with id db7b83dd409ada0309ee08322f020042147c108263047e9353da45c7e0030bab
Aug 01 00:02:45.563977 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:45.563930    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerStarted","Data":"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"}
Aug 01 00:02:45.564504 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:45.563991    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerStarted","Data":"db7b83dd409ada0309ee08322f020042147c108263047e9353da45c7e0030bab"}
Aug 01 00:02:49.562609 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:49.562575    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:02:49.563187 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:49.563160    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:02:49.587612 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:49.587577    2536 generic.go:356] "Generic (PLEG): container finished" podID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerID="57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8" exitCode=0
Aug 01 00:02:49.587787 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:49.587628    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerDied","Data":"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"}
Aug 01 00:02:50.595551 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:50.595515    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerStarted","Data":"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"}
Aug 01 00:02:50.595962 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:50.595582    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerStarted","Data":"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"}
Aug 01 00:02:50.595962 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:50.595845    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:50.615359 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:50.615301    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podStartSLOduration=2.771308554 podStartE2EDuration="6.615281444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84397289s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:02:50.61355509 +0000 UTC m=+995.578750705" watchObservedRunningTime="2026-08-01 00:02:50.615281444 +0000 UTC m=+995.580477067"
Aug 01 00:02:51.600520 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:51.600472    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:51.601614 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:51.601554    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:02:52.605941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:52.605896    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:02:57.611165 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:57.611132    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:02:57.611705 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:57.611675    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:02:59.563716 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:02:59.563669    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:07.611925 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:07.611878    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:09.563586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:09.563546    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:17.612649 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:17.612605    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:19.564095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:19.564055    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:27.612619 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:27.612573    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:29.563801 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:29.563605    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:37.611756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:37.611704    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:39.563916 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:39.563865    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:47.611822 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:47.611774    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:49.563201 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:49.563154    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:56.556238 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:56.556183    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:03:57.612313 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:57.612264    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:03:58.555917 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:03:58.555874    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:04:06.556475 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:06.556422    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:04:08.556560 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:08.556452    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:04:14.478595 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:14.478548    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:04:14.479087 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:14.478884    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" containerID="cri-o://b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467" gracePeriod=30
Aug 01 00:04:14.479155 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:14.479087    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kube-rbac-proxy" containerID="cri-o://d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827" gracePeriod=30
Aug 01 00:04:14.500878 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:14.500829    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:15.036241 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:15.036203    2536 generic.go:356] "Generic (PLEG): container finished" podID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerID="d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827" exitCode=2
Aug 01 00:04:15.036241 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:15.036242    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerDied","Data":"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"}
Aug 01 00:04:16.556498 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:16.556448    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:04:17.606996 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:17.606943    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 01 00:04:18.556030 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:18.555967    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 01 00:04:19.613918 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.613846    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:19.653053 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.653019    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:04:19.762205 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.762087    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:19.779114 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779078    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config\") pod \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") "
Aug 01 00:04:19.779271 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779135    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert\") pod \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") "
Aug 01 00:04:19.779271 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779159    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location\") pod \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") "
Aug 01 00:04:19.779271 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779178    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2q7s\" (UniqueName: \"kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s\") pod \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") "
Aug 01 00:04:19.779271 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779223    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls\") pod \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\" (UID: \"f2f2cb58-9568-4981-811f-2ce5fdebe58f\") "
Aug 01 00:04:19.779556 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779530    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2f2cb58-9568-4981-811f-2ce5fdebe58f" (UID: "f2f2cb58-9568-4981-811f-2ce5fdebe58f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:19.779637 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779575    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2f2cb58-9568-4981-811f-2ce5fdebe58f" (UID: "f2f2cb58-9568-4981-811f-2ce5fdebe58f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:19.779637 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.779615    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2f2cb58-9568-4981-811f-2ce5fdebe58f" (UID: "f2f2cb58-9568-4981-811f-2ce5fdebe58f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:19.781433 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.781408    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2f2cb58-9568-4981-811f-2ce5fdebe58f" (UID: "f2f2cb58-9568-4981-811f-2ce5fdebe58f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:19.781532 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.781452    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s" (OuterVolumeSpecName: "kube-api-access-p2q7s") pod "f2f2cb58-9568-4981-811f-2ce5fdebe58f" (UID: "f2f2cb58-9568-4981-811f-2ce5fdebe58f"). InnerVolumeSpecName "kube-api-access-p2q7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:19.879894 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.879848    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:04:19.879894 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.879887    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:04:19.879894 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.879900    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2q7s\" (UniqueName: \"kubernetes.io/projected/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-api-access-p2q7s\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:04:19.880151 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.879912    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f2cb58-9568-4981-811f-2ce5fdebe58f-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:04:19.880151 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:19.879925    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f2cb58-9568-4981-811f-2ce5fdebe58f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:04:20.063742 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.063704    2536 generic.go:356] "Generic (PLEG): container finished" podID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerID="b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467" exitCode=0
Aug 01 00:04:20.063958 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.063778    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"
Aug 01 00:04:20.063958 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.063791    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerDied","Data":"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"}
Aug 01 00:04:20.063958 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.063857    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl" event={"ID":"f2f2cb58-9568-4981-811f-2ce5fdebe58f","Type":"ContainerDied","Data":"db7b83dd409ada0309ee08322f020042147c108263047e9353da45c7e0030bab"}
Aug 01 00:04:20.063958 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.063886    2536 scope.go:117] "RemoveContainer" containerID="d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"
Aug 01 00:04:20.073146 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.073125    2536 scope.go:117] "RemoveContainer" containerID="b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"
Aug 01 00:04:20.083347 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.082922    2536 scope.go:117] "RemoveContainer" containerID="57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"
Aug 01 00:04:20.086696 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.086672    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:04:20.091945 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.091915    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-7288f-predictor-799b9b69d-r56bl"]
Aug 01 00:04:20.094427 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.094409    2536 scope.go:117] "RemoveContainer" containerID="d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"
Aug 01 00:04:20.094827 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:04:20.094808    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827\": container with ID starting with d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827 not found: ID does not exist" containerID="d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"
Aug 01 00:04:20.094897 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.094836    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827"} err="failed to get container status \"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827\": rpc error: code = NotFound desc = could not find container \"d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827\": container with ID starting with d54d01f0c3473b73e277777d468b3e2e94a0658e130b197dd3139a4d855bc827 not found: ID does not exist"
Aug 01 00:04:20.094897 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.094856    2536 scope.go:117] "RemoveContainer" containerID="b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"
Aug 01 00:04:20.095110 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:04:20.095095    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467\": container with ID starting with b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467 not found: ID does not exist" containerID="b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"
Aug 01 00:04:20.095156 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.095113    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467"} err="failed to get container status \"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467\": rpc error: code = NotFound desc = could not find container \"b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467\": container with ID starting with b6240801f4c09781ff13ca1dcdd1dedc9944f1ffd4bc687d12e31e233237e467 not found: ID does not exist"
Aug 01 00:04:20.095156 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.095125    2536 scope.go:117] "RemoveContainer" containerID="57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"
Aug 01 00:04:20.095359 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:04:20.095340    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8\": container with ID starting with 57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8 not found: ID does not exist" containerID="57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"
Aug 01 00:04:20.095418 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:20.095371    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8"} err="failed to get container status \"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8\": rpc error: code = NotFound desc = could not find container \"57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8\": container with ID starting with 57e229fa08f2652d3fbfbfd7cd5c54c15d2b3914ae6b2830e9caf6f5c56244f8 not found: ID does not exist"
Aug 01 00:04:21.561699 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:21.561661    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" path="/var/lib/kubelet/pods/f2f2cb58-9568-4981-811f-2ce5fdebe58f/volumes"
Aug 01 00:04:26.556156 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:26.556100    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:04:36.556136 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:36.556087    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:04:46.556928 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:46.556870    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:04:56.556597 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:04:56.556543    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:05:06.556842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:06.556783    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:05:16.556315 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:16.556257    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:05:24.556504 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:24.556451    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:05:26.341708 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:26.341668    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:05:26.361807 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:26.361772    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:26.382965 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:26.382931    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" containerID="cri-o://a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e" gracePeriod=30
Aug 01 00:05:26.383133 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:26.382974    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy" containerID="cri-o://e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46" gracePeriod=30
Aug 01 00:05:27.389443 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:27.389413    2536 generic.go:356] "Generic (PLEG): container finished" podID="a2834430-a979-475c-993d-b9ddc36b85b0" containerID="e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46" exitCode=2
Aug 01 00:05:27.389866 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:27.389518    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerDied","Data":"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"}
Aug 01 00:05:29.558538 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:29.558474    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Aug 01 00:05:31.508139 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.508102    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:31.569010 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.568978    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:05:31.569365 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569345    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kube-rbac-proxy"
Aug 01 00:05:31.569365 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569363    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kube-rbac-proxy"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569377    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569382    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569394    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="storage-initializer"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569400    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="storage-initializer"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569460    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kserve-container"
Aug 01 00:05:31.569539 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.569468    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f2cb58-9568-4981-811f-2ce5fdebe58f" containerName="kube-rbac-proxy"
Aug 01 00:05:31.572988 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.572969    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.574987 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.574968    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Aug 01 00:05:31.575130 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.575109    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Aug 01 00:05:31.583075 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.583042    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:05:31.583985 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.583958    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:31.606182 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.606149    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwd6l\" (UniqueName: \"kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.606319 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.606215    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.606319 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.606233    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.606319 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.606274    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.606434 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.606331    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707666 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.707631    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707847 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.707675    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707847 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.707701    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd6l\" (UniqueName: \"kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707847 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.707772    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707847 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.707798    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.707847 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:05:31.707811    2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Aug 01 00:05:31.708124 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:05:31.707892    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls podName:b08c6a53-a7fe-475c-a981-4a088d8f3523 nodeName:}" failed. No retries permitted until 2026-08-01 00:05:32.20787051 +0000 UTC m=+1157.173066133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls") pod "isvc-canary-create-v2-predictor-c4c849674-t6cct" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Aug 01 00:05:31.708249 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.708218    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.708410 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.708390    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.708515 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.708406    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.715567 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.715544    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd6l\" (UniqueName: \"kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:31.786968 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.786933    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:31.910592 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:31.910534    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:32.211706 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:32.211623    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:32.213979 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:32.213958    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-t6cct\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:32.484255 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:32.484164    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:32.620614 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:32.620588    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:05:32.622425 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:05:32.622398    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08c6a53_a7fe_475c_a981_4a088d8f3523.slice/crio-ce732f1e029f03d6d0eb45c8af497a766a18f72d50a4404e699eabe9547c4232 WatchSource:0}: Error finding container ce732f1e029f03d6d0eb45c8af497a766a18f72d50a4404e699eabe9547c4232: Status 404 returned error can't find the container with id ce732f1e029f03d6d0eb45c8af497a766a18f72d50a4404e699eabe9547c4232
Aug 01 00:05:33.419679 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:33.419643    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerStarted","Data":"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"}
Aug 01 00:05:33.419859 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:33.419701    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerStarted","Data":"ce732f1e029f03d6d0eb45c8af497a766a18f72d50a4404e699eabe9547c4232"}
Aug 01 00:05:34.556640 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:34.556592    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 01 00:05:34.557898 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:34.557869    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Aug 01 00:05:36.327952 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.327928    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:05:36.348578 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.348554    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location\") pod \"a2834430-a979-475c-993d-b9ddc36b85b0\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") "
Aug 01 00:05:36.348739 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.348640    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config\") pod \"a2834430-a979-475c-993d-b9ddc36b85b0\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") "
Aug 01 00:05:36.348739 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.348681    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbl4p\" (UniqueName: \"kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p\") pod \"a2834430-a979-475c-993d-b9ddc36b85b0\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") "
Aug 01 00:05:36.348739 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.348714    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") pod \"a2834430-a979-475c-993d-b9ddc36b85b0\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") "
Aug 01 00:05:36.348901 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.348815    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert\") pod \"a2834430-a979-475c-993d-b9ddc36b85b0\" (UID: \"a2834430-a979-475c-993d-b9ddc36b85b0\") "
Aug 01 00:05:36.349065 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349024    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2834430-a979-475c-993d-b9ddc36b85b0" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:05:36.349065 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349014    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2834430-a979-475c-993d-b9ddc36b85b0" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:36.349373 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349361    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2834430-a979-475c-993d-b9ddc36b85b0" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:05:36.349677 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349654    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:05:36.349772 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349682    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2834430-a979-475c-993d-b9ddc36b85b0-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:05:36.349772 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.349696    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2834430-a979-475c-993d-b9ddc36b85b0-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:05:36.351528 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.351453    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p" (OuterVolumeSpecName: "kube-api-access-sbl4p") pod "a2834430-a979-475c-993d-b9ddc36b85b0" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0"). InnerVolumeSpecName "kube-api-access-sbl4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:05:36.351657 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.351566    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2834430-a979-475c-993d-b9ddc36b85b0" (UID: "a2834430-a979-475c-993d-b9ddc36b85b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:05:36.435566 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.435530    2536 generic.go:356] "Generic (PLEG): container finished" podID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerID="a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba" exitCode=0
Aug 01 00:05:36.435738 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.435600    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerDied","Data":"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"}
Aug 01 00:05:36.437618 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.437595    2536 generic.go:356] "Generic (PLEG): container finished" podID="a2834430-a979-475c-993d-b9ddc36b85b0" containerID="a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e" exitCode=0
Aug 01 00:05:36.437721 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.437625    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerDied","Data":"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"}
Aug 01 00:05:36.437721 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.437640    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"
Aug 01 00:05:36.437721 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.437685    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d" event={"ID":"a2834430-a979-475c-993d-b9ddc36b85b0","Type":"ContainerDied","Data":"612632f37fdd2137157022b1f1117616d167133693ce818d3e79d2361e3687d7"}
Aug 01 00:05:36.437930 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.437722    2536 scope.go:117] "RemoveContainer" containerID="e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"
Aug 01 00:05:36.448897 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.448879    2536 scope.go:117] "RemoveContainer" containerID="a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"
Aug 01 00:05:36.451126 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.451102    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbl4p\" (UniqueName: \"kubernetes.io/projected/a2834430-a979-475c-993d-b9ddc36b85b0-kube-api-access-sbl4p\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:05:36.451285 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.451128    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2834430-a979-475c-993d-b9ddc36b85b0-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:05:36.457985 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.457960    2536 scope.go:117] "RemoveContainer" containerID="2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"
Aug 01 00:05:36.467244 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.467205    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:05:36.473304 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.473075    2536 scope.go:117] "RemoveContainer" containerID="e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"
Aug 01 00:05:36.473743 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:05:36.473717    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46\": container with ID starting with e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46 not found: ID does not exist" containerID="e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"
Aug 01 00:05:36.473848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.473753    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46"} err="failed to get container status \"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46\": rpc error: code = NotFound desc = could not find container \"e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46\": container with ID starting with e207f81cc6e8da745134ed85441b2f3d4dbf76ad89042a276912faf641bb0b46 not found: ID does not exist"
Aug 01 00:05:36.473848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.473779    2536 scope.go:117] "RemoveContainer" containerID="a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"
Aug 01 00:05:36.474110 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:05:36.474089    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e\": container with ID starting with a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e not found: ID does not exist" containerID="a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"
Aug 01 00:05:36.474214 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.474121    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e"} err="failed to get container status \"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e\": rpc error: code = NotFound desc = could not find container \"a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e\": container with ID starting with a9681cfbfb30d43258f4c39dea000f3aa8d61e2af261502af93d79c213cd3a7e not found: ID does not exist"
Aug 01 00:05:36.474214 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.474163    2536 scope.go:117] "RemoveContainer" containerID="2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"
Aug 01 00:05:36.475835 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:05:36.474604    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40\": container with ID starting with 2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40 not found: ID does not exist" containerID="2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"
Aug 01 00:05:36.475835 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.474645    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40"} err="failed to get container status \"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40\": rpc error: code = NotFound desc = could not find container \"2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40\": container with ID starting with 2a507107de29cdc9362628d747cae6db0898b797370b4a9658680b6af4f4db40 not found: ID does not exist"
Aug 01 00:05:36.476055 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:36.475961    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-83d7e-predictor-776f858d64-p828d"]
Aug 01 00:05:37.445295 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:37.445258    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerStarted","Data":"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"}
Aug 01 00:05:37.445821 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:37.445359    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerStarted","Data":"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"}
Aug 01 00:05:37.445821 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:37.445588    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:37.464706 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:37.464655    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podStartSLOduration=2.897913746 podStartE2EDuration="6.464639969s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.566726223s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:05:37.462801249 +0000 UTC m=+1162.427996883" watchObservedRunningTime="2026-08-01 00:05:37.464639969 +0000 UTC m=+1162.429835647"
Aug 01 00:05:37.561243 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:37.561209    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" path="/var/lib/kubelet/pods/a2834430-a979-475c-993d-b9ddc36b85b0/volumes"
Aug 01 00:05:38.449828 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:38.449799    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:38.451254 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:38.451221    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:05:39.454479 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:39.454440    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:05:44.458912 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:44.458884    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:05:44.459377 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:44.459349    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:05:49.816397 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:49.816362    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:51.145338 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:51.145302    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:54.459820 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:05:54.459781    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:04.459627 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:04.459582    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:14.459832 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:14.459792    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:15.569378 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:15.569350    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:06:15.571779 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:15.571757    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:06:24.459651 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:24.459611    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:34.459546 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:34.459474    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:44.459858 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:44.459815    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:06:54.460597 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:06:54.460568    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:07:01.621950 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.621912    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:07:01.622586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.622171    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" containerID="cri-o://e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3" gracePeriod=30
Aug 01 00:07:01.622586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.622389    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kube-rbac-proxy" containerID="cri-o://491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e" gracePeriod=30
Aug 01 00:07:01.631195 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.631169    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:01.731734 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.731699    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:01.856882 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.856851    2536 generic.go:356] "Generic (PLEG): container finished" podID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerID="491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e" exitCode=2
Aug 01 00:07:01.857063 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:01.856923    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerDied","Data":"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"}
Aug 01 00:07:04.455218 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:04.455170    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Aug 01 00:07:04.459456 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:04.459424    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Aug 01 00:07:06.376236 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.376209    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:07:06.544218 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544180    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") pod \"b08c6a53-a7fe-475c-a981-4a088d8f3523\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") "
Aug 01 00:07:06.544417 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544225    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config\") pod \"b08c6a53-a7fe-475c-a981-4a088d8f3523\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") "
Aug 01 00:07:06.544417 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544246    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwd6l\" (UniqueName: \"kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l\") pod \"b08c6a53-a7fe-475c-a981-4a088d8f3523\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") "
Aug 01 00:07:06.544417 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544306    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert\") pod \"b08c6a53-a7fe-475c-a981-4a088d8f3523\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") "
Aug 01 00:07:06.544417 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544369    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location\") pod \"b08c6a53-a7fe-475c-a981-4a088d8f3523\" (UID: \"b08c6a53-a7fe-475c-a981-4a088d8f3523\") "
Aug 01 00:07:06.544728 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544663    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b08c6a53-a7fe-475c-a981-4a088d8f3523" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:06.544861 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544748    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b08c6a53-a7fe-475c-a981-4a088d8f3523" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:06.544861 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.544784    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b08c6a53-a7fe-475c-a981-4a088d8f3523" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:07:06.546384 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.546362    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b08c6a53-a7fe-475c-a981-4a088d8f3523" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:07:06.546475 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.546411    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l" (OuterVolumeSpecName: "kube-api-access-kwd6l") pod "b08c6a53-a7fe-475c-a981-4a088d8f3523" (UID: "b08c6a53-a7fe-475c-a981-4a088d8f3523"). InnerVolumeSpecName "kube-api-access-kwd6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:07:06.645500 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.645461    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08c6a53-a7fe-475c-a981-4a088d8f3523-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:07:06.645737 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.645515    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08c6a53-a7fe-475c-a981-4a088d8f3523-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:07:06.645737 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.645533    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:07:06.645737 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.645547    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwd6l\" (UniqueName: \"kubernetes.io/projected/b08c6a53-a7fe-475c-a981-4a088d8f3523-kube-api-access-kwd6l\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:07:06.645737 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.645562    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08c6a53-a7fe-475c-a981-4a088d8f3523-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:07:06.779226 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.779181    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:06.882365 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882276    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:07:06.882616 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882587    2536 generic.go:356] "Generic (PLEG): container finished" podID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerID="e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3" exitCode=0
Aug 01 00:07:06.882757 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882731    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"
Aug 01 00:07:06.882803 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882788    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="storage-initializer"
Aug 01 00:07:06.882848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882805    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="storage-initializer"
Aug 01 00:07:06.882848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882819    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container"
Aug 01 00:07:06.882848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882830    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882852    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882859    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882878    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882884    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882889    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kube-rbac-proxy"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882894    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kube-rbac-proxy"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882905    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="storage-initializer"
Aug 01 00:07:06.882973 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.882910    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="storage-initializer"
Aug 01 00:07:06.883256 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.883090    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kserve-container"
Aug 01 00:07:06.883256 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.883102    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" containerName="kube-rbac-proxy"
Aug 01 00:07:06.883256 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.883109    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kube-rbac-proxy"
Aug 01 00:07:06.883256 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.883115    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2834430-a979-475c-993d-b9ddc36b85b0" containerName="kserve-container"
Aug 01 00:07:06.886646 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.886623    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerDied","Data":"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"}
Aug 01 00:07:06.886742 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.886661    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct" event={"ID":"b08c6a53-a7fe-475c-a981-4a088d8f3523","Type":"ContainerDied","Data":"ce732f1e029f03d6d0eb45c8af497a766a18f72d50a4404e699eabe9547c4232"}
Aug 01 00:07:06.886742 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.886694    2536 scope.go:117] "RemoveContainer" containerID="491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"
Aug 01 00:07:06.886742 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.886738    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:06.893007 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.892986    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Aug 01 00:07:06.893120 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.892993    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:07:06.893354 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.893334    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 01 00:07:06.893454 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.893334    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:07:06.893454 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.893334    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:07:06.893454 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.893419    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wctjk\""
Aug 01 00:07:06.897041 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.897022    2536 scope.go:117] "RemoveContainer" containerID="e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"
Aug 01 00:07:06.898154 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.898138    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:07:06.902126 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.902106    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:07:06.906140 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.906122    2536 scope.go:117] "RemoveContainer" containerID="a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"
Aug 01 00:07:06.907082 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.907067    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:06.909670 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.909651    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 01 00:07:06.915234 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.915219    2536 scope.go:117] "RemoveContainer" containerID="491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"
Aug 01 00:07:06.915605 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:07:06.915580    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e\": container with ID starting with 491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e not found: ID does not exist" containerID="491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"
Aug 01 00:07:06.915665 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.915616    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e"} err="failed to get container status \"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e\": rpc error: code = NotFound desc = could not find container \"491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e\": container with ID starting with 491c94c3bd815fd9771296ee351959bd1e93d0cd9da028c8dce1bac162bfc88e not found: ID does not exist"
Aug 01 00:07:06.915665 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.915637    2536 scope.go:117] "RemoveContainer" containerID="e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"
Aug 01 00:07:06.915909 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:07:06.915879    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3\": container with ID starting with e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3 not found: ID does not exist" containerID="e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"
Aug 01 00:07:06.916006 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.915917    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3"} err="failed to get container status \"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3\": rpc error: code = NotFound desc = could not find container \"e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3\": container with ID starting with e109e972041e76b2b1ed294135fa44da7f3aa15172051abc2ec9d86f391effc3 not found: ID does not exist"
Aug 01 00:07:06.916006 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.915941    2536 scope.go:117] "RemoveContainer" containerID="a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"
Aug 01 00:07:06.916185 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:07:06.916168    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba\": container with ID starting with a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba not found: ID does not exist" containerID="a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"
Aug 01 00:07:06.916230 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.916192    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba"} err="failed to get container status \"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba\": rpc error: code = NotFound desc = could not find container \"a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba\": container with ID starting with a418d3ab885e57be051233b96701854f7c55aa9eb9fb7b8e66ae224519bce6ba not found: ID does not exist"
Aug 01 00:07:06.926345 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.926318    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:07:06.942718 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.942692    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:07:06.962033 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.961998    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:06.996522 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:06.996470    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:07:07.004468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.004435    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-t6cct"]
Aug 01 00:07:07.050617 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050581    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.050617 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050620    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxt7\" (UniqueName: \"kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.050832 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050722    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kd5\" (UniqueName: \"kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.050832 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050752    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.050832 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050773    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.050832 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050825    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.050960 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050863    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.050960 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050887    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.050960 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050912    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.050960 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.050930    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.061238 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.061201    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:07.151446 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151358    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.151446 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151420    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151447    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxt7\" (UniqueName: \"kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151539    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kd5\" (UniqueName: \"kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151569    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151597    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151628    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151680    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.151802 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151704    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.152135 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.151808    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.152135 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:07:07.151920    2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:07:07.152135 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:07:07.152020    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls podName:2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68 nodeName:}" failed. No retries permitted until 2026-08-01 00:07:07.651997924 +0000 UTC m=+1252.617193547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls") pod "isvc-canary-promote-predictor-569b87b6dc-sbfmj" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68") : secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:07:07.152314 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152142    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.152375 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152358    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.152450 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152419    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.152600 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152577    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.152823 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152799    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.152907 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.152808    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.154535 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.154518    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.166135 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.166107    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kd5\" (UniqueName: \"kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.166135 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.166117    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxt7\" (UniqueName: \"kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.217843 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.217812    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:07.357273 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.357249    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:07:07.359619 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:07:07.359587    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60a45f0_4443_4dc1_b200_b07f25a9c2d6.slice/crio-56e55e72a592310b268d3aa3652685ce03c3707acb9eefe840b7c252e8265d18 WatchSource:0}: Error finding container 56e55e72a592310b268d3aa3652685ce03c3707acb9eefe840b7c252e8265d18: Status 404 returned error can't find the container with id 56e55e72a592310b268d3aa3652685ce03c3707acb9eefe840b7c252e8265d18
Aug 01 00:07:07.561678 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.561637    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08c6a53-a7fe-475c-a981-4a088d8f3523" path="/var/lib/kubelet/pods/b08c6a53-a7fe-475c-a981-4a088d8f3523/volumes"
Aug 01 00:07:07.587778 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.587738    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:07.655106 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.655010    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.657796 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.657766    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-sbfmj\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.800347 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.800263    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:07.891837 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.891803    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerStarted","Data":"e9ae0776d9b41c4c3358e45e63a58fd4baf38dd9c7763e5a316ea076b06092cc"}
Aug 01 00:07:07.891971 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.891877    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerStarted","Data":"56e55e72a592310b268d3aa3652685ce03c3707acb9eefe840b7c252e8265d18"}
Aug 01 00:07:07.949619 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:07.949590    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:07:07.950059 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:07:07.950036    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2befb9f0_d40e_4c44_9b4c_e0b9b19f7b68.slice/crio-6146bfd53dc41dd9979fce9077313461dd702fd27fbf7d5562c66e19dde79ada WatchSource:0}: Error finding container 6146bfd53dc41dd9979fce9077313461dd702fd27fbf7d5562c66e19dde79ada: Status 404 returned error can't find the container with id 6146bfd53dc41dd9979fce9077313461dd702fd27fbf7d5562c66e19dde79ada
Aug 01 00:07:08.898000 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:08.897959    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerStarted","Data":"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"}
Aug 01 00:07:08.898475 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:08.898027    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerStarted","Data":"6146bfd53dc41dd9979fce9077313461dd702fd27fbf7d5562c66e19dde79ada"}
Aug 01 00:07:11.913095 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:11.913052    2536 generic.go:356] "Generic (PLEG): container finished" podID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerID="e9ae0776d9b41c4c3358e45e63a58fd4baf38dd9c7763e5a316ea076b06092cc" exitCode=0
Aug 01 00:07:11.913641 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:11.913115    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerDied","Data":"e9ae0776d9b41c4c3358e45e63a58fd4baf38dd9c7763e5a316ea076b06092cc"}
Aug 01 00:07:11.914956 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:11.914931    2536 generic.go:356] "Generic (PLEG): container finished" podID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerID="3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6" exitCode=0
Aug 01 00:07:11.915031 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:11.915002    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerDied","Data":"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"}
Aug 01 00:07:12.923064 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.923031    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerStarted","Data":"d23945abbb53943e741597b3b508d90417933665da88693d57397d6c961bb78c"}
Aug 01 00:07:12.923558 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.923103    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerStarted","Data":"b0f648b1993c4525e2ad694d27b628bce6d24f2e37524310a0bd7498feb36d59"}
Aug 01 00:07:12.923558 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.923392    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:12.923558 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.923553    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:12.924867 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.924832    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:12.925326 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.925308    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerStarted","Data":"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"}
Aug 01 00:07:12.925402 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.925350    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerStarted","Data":"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"}
Aug 01 00:07:12.925591 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.925574    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:12.925666 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.925615    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:12.926650 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.926607    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:12.943639 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.943585    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podStartSLOduration=3.420215247 podStartE2EDuration="6.943566768s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.523351521s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:12.942981133 +0000 UTC m=+1257.908176758" watchObservedRunningTime="2026-08-01 00:07:12.943566768 +0000 UTC m=+1257.908762391"
Aug 01 00:07:12.963038 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:12.962990    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podStartSLOduration=3.285676295 podStartE2EDuration="6.962961616s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677285321s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:12.961962776 +0000 UTC m=+1257.927158411" watchObservedRunningTime="2026-08-01 00:07:12.962961616 +0000 UTC m=+1257.928157249"
Aug 01 00:07:13.930500 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:13.930437    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:13.930879 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:13.930544    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:18.934781 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:18.934699    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:07:18.935224 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:18.934996    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:07:18.935353 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:18.935326    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:18.935693 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:18.935666    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:28.935745 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:28.935701    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:28.936123 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:28.935698    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:38.935982 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:38.935939    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:38.936519 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:38.935939    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:39.061724 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:39.061688    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:48.935423 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:48.935369    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:48.935902 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:48.935560    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:07:58.935811 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:58.935768    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:07:58.936233 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:07:58.935768    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:08:08.935875 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:08.935841    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:08:08.936352 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:08.935836    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:08:18.935563 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:18.935511    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:08:18.936044 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:18.935649    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:08:28.935639 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:28.935609    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:08:28.936137 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:28.936089    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:08:29.152085 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:29.152055    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:36.903796 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:36.903763    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:08:36.904217 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:36.904075    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" containerID="cri-o://ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e" gracePeriod=30
Aug 01 00:08:36.904217 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:36.904100    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kube-rbac-proxy" containerID="cri-o://d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e" gracePeriod=30
Aug 01 00:08:36.927311 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:36.927280    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:37.343544 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:37.343481    2536 generic.go:356] "Generic (PLEG): container finished" podID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerID="d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e" exitCode=2
Aug 01 00:08:37.343722 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:37.343551    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerDied","Data":"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"}
Aug 01 00:08:38.930612 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:38.930566    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Aug 01 00:08:38.935583 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:38.935552    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 01 00:08:41.658184 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.658153    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:08:41.787018 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.786985    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert\") pod \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") "
Aug 01 00:08:41.787192 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787042    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location\") pod \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") "
Aug 01 00:08:41.787192 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787098    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") pod \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") "
Aug 01 00:08:41.787192 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787127    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5kd5\" (UniqueName: \"kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5\") pod \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") "
Aug 01 00:08:41.787192 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787151    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config\") pod \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\" (UID: \"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68\") "
Aug 01 00:08:41.787427 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787347    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:41.787513 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787431    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:08:41.787513 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787432    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:08:41.787599 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.787565    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:41.789193 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.789170    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:41.789290 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.789211    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5" (OuterVolumeSpecName: "kube-api-access-f5kd5") pod "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" (UID: "2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68"). InnerVolumeSpecName "kube-api-access-f5kd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:41.888588 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.888549    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:08:41.888588 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.888580    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:08:41.888588 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.888591    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5kd5\" (UniqueName: \"kubernetes.io/projected/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-api-access-f5kd5\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:08:41.888588 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:41.888599    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:08:42.368682 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.368647    2536 generic.go:356] "Generic (PLEG): container finished" podID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerID="ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e" exitCode=0
Aug 01 00:08:42.368889 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.368731    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"
Aug 01 00:08:42.368889 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.368728    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerDied","Data":"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"}
Aug 01 00:08:42.368889 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.368865    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj" event={"ID":"2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68","Type":"ContainerDied","Data":"6146bfd53dc41dd9979fce9077313461dd702fd27fbf7d5562c66e19dde79ada"}
Aug 01 00:08:42.369040 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.368894    2536 scope.go:117] "RemoveContainer" containerID="d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"
Aug 01 00:08:42.378629 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.378612    2536 scope.go:117] "RemoveContainer" containerID="ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"
Aug 01 00:08:42.386543 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.386479    2536 scope.go:117] "RemoveContainer" containerID="3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"
Aug 01 00:08:42.393072 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.393048    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:08:42.394719 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.394701    2536 scope.go:117] "RemoveContainer" containerID="d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"
Aug 01 00:08:42.394982 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:08:42.394964    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e\": container with ID starting with d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e not found: ID does not exist" containerID="d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"
Aug 01 00:08:42.395052 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.394991    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e"} err="failed to get container status \"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e\": rpc error: code = NotFound desc = could not find container \"d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e\": container with ID starting with d21bf11d51a95359f89d98f46d3dfa3586d575a51c6bcce4a20bb71f77ce6a1e not found: ID does not exist"
Aug 01 00:08:42.395052 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.395011    2536 scope.go:117] "RemoveContainer" containerID="ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"
Aug 01 00:08:42.395302 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:08:42.395273    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e\": container with ID starting with ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e not found: ID does not exist" containerID="ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"
Aug 01 00:08:42.395426 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.395305    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e"} err="failed to get container status \"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e\": rpc error: code = NotFound desc = could not find container \"ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e\": container with ID starting with ccc9b12b0ea8610cb4b18258052bbfe210a02e3d3d7b18518a744c61ca87342e not found: ID does not exist"
Aug 01 00:08:42.395426 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.395323    2536 scope.go:117] "RemoveContainer" containerID="3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"
Aug 01 00:08:42.395806 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:08:42.395776    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6\": container with ID starting with 3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6 not found: ID does not exist" containerID="3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"
Aug 01 00:08:42.395897 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.395810    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6"} err="failed to get container status \"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6\": rpc error: code = NotFound desc = could not find container \"3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6\": container with ID starting with 3e46cbf8c7f6cfbd7826faf28d15b5653826ea30bb7b96535d9645dff6a4a7c6 not found: ID does not exist"
Aug 01 00:08:42.399208 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:42.399187    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-sbfmj"]
Aug 01 00:08:43.561918 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:08:43.561885    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" path="/var/lib/kubelet/pods/2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68/volumes"
Aug 01 00:09:54.486221 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:09:54.486186    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:05.720508 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:05.720455    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:07.135745 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.135709    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:10:07.136301 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.136266    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" containerID="cri-o://b0f648b1993c4525e2ad694d27b628bce6d24f2e37524310a0bd7498feb36d59" gracePeriod=30
Aug 01 00:10:07.136404 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.136306    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kube-rbac-proxy" containerID="cri-o://d23945abbb53943e741597b3b508d90417933665da88693d57397d6c961bb78c" gracePeriod=30
Aug 01 00:10:07.163029 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.162995    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:07.787378 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.787303    2536 generic.go:356] "Generic (PLEG): container finished" podID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerID="d23945abbb53943e741597b3b508d90417933665da88693d57397d6c961bb78c" exitCode=2
Aug 01 00:10:07.787555 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:07.787376    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerDied","Data":"d23945abbb53943e741597b3b508d90417933665da88693d57397d6c961bb78c"}
Aug 01 00:10:08.930996 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:08.930949    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Aug 01 00:10:08.936313 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:08.936284    2536 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 01 00:10:11.808814 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:11.808781    2536 generic.go:356] "Generic (PLEG): container finished" podID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerID="b0f648b1993c4525e2ad694d27b628bce6d24f2e37524310a0bd7498feb36d59" exitCode=0
Aug 01 00:10:11.809170 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:11.808833    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerDied","Data":"b0f648b1993c4525e2ad694d27b628bce6d24f2e37524310a0bd7498feb36d59"}
Aug 01 00:10:11.893932 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:11.893903    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:10:12.001054 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.000962    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls\") pod \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") "
Aug 01 00:10:12.001054 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001015    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bxt7\" (UniqueName: \"kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7\") pod \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") "
Aug 01 00:10:12.001054 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001039    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert\") pod \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") "
Aug 01 00:10:12.001348 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001088    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location\") pod \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") "
Aug 01 00:10:12.001348 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001168    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config\") pod \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\" (UID: \"e60a45f0-4443-4dc1-b200-b07f25a9c2d6\") "
Aug 01 00:10:12.001554 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001522    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e60a45f0-4443-4dc1-b200-b07f25a9c2d6" (UID: "e60a45f0-4443-4dc1-b200-b07f25a9c2d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:12.001554 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001541    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e60a45f0-4443-4dc1-b200-b07f25a9c2d6" (UID: "e60a45f0-4443-4dc1-b200-b07f25a9c2d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:12.001659 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.001589    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e60a45f0-4443-4dc1-b200-b07f25a9c2d6" (UID: "e60a45f0-4443-4dc1-b200-b07f25a9c2d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:12.003193 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.003167    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e60a45f0-4443-4dc1-b200-b07f25a9c2d6" (UID: "e60a45f0-4443-4dc1-b200-b07f25a9c2d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:12.003273 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.003219    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7" (OuterVolumeSpecName: "kube-api-access-4bxt7") pod "e60a45f0-4443-4dc1-b200-b07f25a9c2d6" (UID: "e60a45f0-4443-4dc1-b200-b07f25a9c2d6"). InnerVolumeSpecName "kube-api-access-4bxt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:12.102376 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.102341    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxt7\" (UniqueName: \"kubernetes.io/projected/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-api-access-4bxt7\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:10:12.102376 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.102371    2536 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-cabundle-cert\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:10:12.102594 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.102386    2536 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kserve-provision-location\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:10:12.102594 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.102398    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:10:12.102594 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.102407    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60a45f0-4443-4dc1-b200-b07f25a9c2d6-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:10:12.814878 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.814840    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5" event={"ID":"e60a45f0-4443-4dc1-b200-b07f25a9c2d6","Type":"ContainerDied","Data":"56e55e72a592310b268d3aa3652685ce03c3707acb9eefe840b7c252e8265d18"}
Aug 01 00:10:12.814878 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.814867    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"
Aug 01 00:10:12.815472 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.814909    2536 scope.go:117] "RemoveContainer" containerID="d23945abbb53943e741597b3b508d90417933665da88693d57397d6c961bb78c"
Aug 01 00:10:12.823772 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.823755    2536 scope.go:117] "RemoveContainer" containerID="b0f648b1993c4525e2ad694d27b628bce6d24f2e37524310a0bd7498feb36d59"
Aug 01 00:10:12.831393 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.831376    2536 scope.go:117] "RemoveContainer" containerID="e9ae0776d9b41c4c3358e45e63a58fd4baf38dd9c7763e5a316ea076b06092cc"
Aug 01 00:10:12.838382 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.838359    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:10:12.842349 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:12.842324    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-dkgt5"]
Aug 01 00:10:13.561794 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:13.561757    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" path="/var/lib/kubelet/pods/e60a45f0-4443-4dc1-b200-b07f25a9c2d6/volumes"
Aug 01 00:10:21.171023 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.170985    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:21.186676 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.186644    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:10:21.187033 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187020    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187034    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187050    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kube-rbac-proxy"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187056    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kube-rbac-proxy"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187067    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="storage-initializer"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187075    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="storage-initializer"
Aug 01 00:10:21.187089 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187087    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="storage-initializer"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187093    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="storage-initializer"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187099    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kube-rbac-proxy"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187105    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kube-rbac-proxy"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187119    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187124    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187182    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kserve-container"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187190    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="2befb9f0-d40e-4c44-9b4c-e0b9b19f7b68" containerName="kube-rbac-proxy"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187198    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kserve-container"
Aug 01 00:10:21.187288 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.187204    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60a45f0-4443-4dc1-b200-b07f25a9c2d6" containerName="kube-rbac-proxy"
Aug 01 00:10:21.190299 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.190282    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.192798 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.192770    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Aug 01 00:10:21.192941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.192825    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:10:21.192941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.192845    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:10:21.192941 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.192881    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Aug 01 00:10:21.193632 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.193616    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wctjk\""
Aug 01 00:10:21.201819 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.201798    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:10:21.282799 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.282759    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.282986 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.282827    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.282986 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.282865    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwzq\" (UniqueName: \"kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.383963 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.383921    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.383963 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.383966    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwzq\" (UniqueName: \"kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.384170 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.384043    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.384170 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:10:21.384079    2536 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Aug 01 00:10:21.384170 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:10:21.384150    2536 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls podName:b263c37b-9433-48c8-a32b-876754b45493 nodeName:}" failed. No retries permitted until 2026-08-01 00:10:21.884133298 +0000 UTC m=+1446.849328901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" (UID: "b263c37b-9433-48c8-a32b-876754b45493") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Aug 01 00:10:21.384788 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.384769    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.394117 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.394083    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwzq\" (UniqueName: \"kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.442327 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.442251    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:21.887500 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.887461    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:21.889878 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:21.889850    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-v5tkf\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:22.101721 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.101669    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:22.244025 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.243989    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:10:22.247805 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:10:22.247777    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb263c37b_9433_48c8_a32b_876754b45493.slice/crio-d07b0b69d5d05463e54022d275e37e2d3cea6fb533f17733e97a04c20d1f7c85 WatchSource:0}: Error finding container d07b0b69d5d05463e54022d275e37e2d3cea6fb533f17733e97a04c20d1f7c85: Status 404 returned error can't find the container with id d07b0b69d5d05463e54022d275e37e2d3cea6fb533f17733e97a04c20d1f7c85
Aug 01 00:10:22.869372 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.869329    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerStarted","Data":"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"}
Aug 01 00:10:22.869586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.869405    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerStarted","Data":"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"}
Aug 01 00:10:22.869586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.869432    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerStarted","Data":"d07b0b69d5d05463e54022d275e37e2d3cea6fb533f17733e97a04c20d1f7c85"}
Aug 01 00:10:22.869586 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.869464    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:22.889057 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:22.888993    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" podStartSLOduration=1.888977121 podStartE2EDuration="1.888977121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:22.887435444 +0000 UTC m=+1447.852631084" watchObservedRunningTime="2026-08-01 00:10:22.888977121 +0000 UTC m=+1447.854172744"
Aug 01 00:10:23.874704 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:23.874661    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:23.876581 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:23.876560    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:30.888790 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:30.888758    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:10:31.166221 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:31.166145    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:31.316884 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:10:31.316840    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:15.599849 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:11:15.599822    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:11:15.602243 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:11:15.602220    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:12:06.238084 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.238026    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-v5tkf_b263c37b-9433-48c8-a32b-876754b45493/kserve-container/0.log"
Aug 01 00:12:06.508216 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.508135    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:12:06.508420 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.508382    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kserve-container" containerID="cri-o://4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42" gracePeriod=30
Aug 01 00:12:06.508502 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.508428    2536 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kube-rbac-proxy" containerID="cri-o://58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020" gracePeriod=30
Aug 01 00:12:06.758504 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.758421    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:12:06.811673 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.811640    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") pod \"b263c37b-9433-48c8-a32b-876754b45493\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") "
Aug 01 00:12:06.811849 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.811686    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bwzq\" (UniqueName: \"kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq\") pod \"b263c37b-9433-48c8-a32b-876754b45493\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") "
Aug 01 00:12:06.811849 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.811766    2536 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config\") pod \"b263c37b-9433-48c8-a32b-876754b45493\" (UID: \"b263c37b-9433-48c8-a32b-876754b45493\") "
Aug 01 00:12:06.812156 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.812129    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b263c37b-9433-48c8-a32b-876754b45493" (UID: "b263c37b-9433-48c8-a32b-876754b45493"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:12:06.813746 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.813719    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b263c37b-9433-48c8-a32b-876754b45493" (UID: "b263c37b-9433-48c8-a32b-876754b45493"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:12:06.813842 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.813758    2536 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq" (OuterVolumeSpecName: "kube-api-access-5bwzq") pod "b263c37b-9433-48c8-a32b-876754b45493" (UID: "b263c37b-9433-48c8-a32b-876754b45493"). InnerVolumeSpecName "kube-api-access-5bwzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:12:06.913058 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.913023    2536 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b263c37b-9433-48c8-a32b-876754b45493-proxy-tls\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:12:06.913058 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.913052    2536 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwzq\" (UniqueName: \"kubernetes.io/projected/b263c37b-9433-48c8-a32b-876754b45493-kube-api-access-5bwzq\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:12:06.913058 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:06.913062    2536 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b263c37b-9433-48c8-a32b-876754b45493-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-129.ec2.internal\" DevicePath \"\""
Aug 01 00:12:07.353023 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.352988    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:07.361242 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361212    2536 generic.go:356] "Generic (PLEG): container finished" podID="b263c37b-9433-48c8-a32b-876754b45493" containerID="58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020" exitCode=2
Aug 01 00:12:07.361242 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361236    2536 generic.go:356] "Generic (PLEG): container finished" podID="b263c37b-9433-48c8-a32b-876754b45493" containerID="4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42" exitCode=2
Aug 01 00:12:07.361457 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361256    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerDied","Data":"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"}
Aug 01 00:12:07.361457 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361267    2536 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"
Aug 01 00:12:07.361457 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361304    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerDied","Data":"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"}
Aug 01 00:12:07.361457 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361327    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf" event={"ID":"b263c37b-9433-48c8-a32b-876754b45493","Type":"ContainerDied","Data":"d07b0b69d5d05463e54022d275e37e2d3cea6fb533f17733e97a04c20d1f7c85"}
Aug 01 00:12:07.361457 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.361350    2536 scope.go:117] "RemoveContainer" containerID="58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"
Aug 01 00:12:07.370234 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.370217    2536 scope.go:117] "RemoveContainer" containerID="4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"
Aug 01 00:12:07.378168 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378148    2536 scope.go:117] "RemoveContainer" containerID="58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"
Aug 01 00:12:07.378427 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:12:07.378407    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020\": container with ID starting with 58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020 not found: ID does not exist" containerID="58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"
Aug 01 00:12:07.378524 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378440    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"} err="failed to get container status \"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020\": rpc error: code = NotFound desc = could not find container \"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020\": container with ID starting with 58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020 not found: ID does not exist"
Aug 01 00:12:07.378524 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378463    2536 scope.go:117] "RemoveContainer" containerID="4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"
Aug 01 00:12:07.378718 ip-10-0-132-129 kubenswrapper[2536]: E0801 00:12:07.378700    2536 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42\": container with ID starting with 4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42 not found: ID does not exist" containerID="4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"
Aug 01 00:12:07.378759 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378725    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"} err="failed to get container status \"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42\": rpc error: code = NotFound desc = could not find container \"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42\": container with ID starting with 4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42 not found: ID does not exist"
Aug 01 00:12:07.378759 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378743    2536 scope.go:117] "RemoveContainer" containerID="58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"
Aug 01 00:12:07.378950 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378932    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020"} err="failed to get container status \"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020\": rpc error: code = NotFound desc = could not find container \"58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020\": container with ID starting with 58cd43150f12ad1ed6a67c42598dec2046ca296a9b3a998e0330668f0f035020 not found: ID does not exist"
Aug 01 00:12:07.379020 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.378952    2536 scope.go:117] "RemoveContainer" containerID="4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"
Aug 01 00:12:07.379233 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.379164    2536 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42"} err="failed to get container status \"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42\": rpc error: code = NotFound desc = could not find container \"4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42\": container with ID starting with 4ca096c6bf07afc9cdc08b3eefebc2be23fcd3e7e1fa458d89d29cb0ea7f4b42 not found: ID does not exist"
Aug 01 00:12:07.381600 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.381577    2536 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:12:07.385279 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.385255    2536 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-v5tkf"]
Aug 01 00:12:07.411233 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.411207    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:07.561500 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:07.561459    2536 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b263c37b-9433-48c8-a32b-876754b45493" path="/var/lib/kubelet/pods/b263c37b-9433-48c8-a32b-876754b45493/volumes"
Aug 01 00:12:38.364666 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.364624    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:38.523131 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523099    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xgqkp/must-gather-7547d"]
Aug 01 00:12:38.523472 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523460    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kserve-container"
Aug 01 00:12:38.523551 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523474    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kserve-container"
Aug 01 00:12:38.523551 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523509    2536 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kube-rbac-proxy"
Aug 01 00:12:38.523551 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523519    2536 state_mem.go:107] "Deleted CPUSet assignment" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kube-rbac-proxy"
Aug 01 00:12:38.523654 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523575    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kserve-container"
Aug 01 00:12:38.523654 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.523585    2536 memory_manager.go:368] "RemoveStaleState removing state" podUID="b263c37b-9433-48c8-a32b-876754b45493" containerName="kube-rbac-proxy"
Aug 01 00:12:38.527786 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.527765    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.530414 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.530388    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xgqkp\"/\"openshift-service-ca.crt\""
Aug 01 00:12:38.530550 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.530393    2536 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xgqkp\"/\"kube-root-ca.crt\""
Aug 01 00:12:38.530550 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.530388    2536 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xgqkp\"/\"default-dockercfg-kblfc\""
Aug 01 00:12:38.534027 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.534007    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgqkp/must-gather-7547d"]
Aug 01 00:12:38.537990 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.537964    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:38.601888 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.601839    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhm2k\" (UniqueName: \"kubernetes.io/projected/b1bc8942-bca2-45de-9d81-31183bbf0505-kube-api-access-lhm2k\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.602104 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.601934    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1bc8942-bca2-45de-9d81-31183bbf0505-must-gather-output\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.702458 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.702353    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhm2k\" (UniqueName: \"kubernetes.io/projected/b1bc8942-bca2-45de-9d81-31183bbf0505-kube-api-access-lhm2k\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.702458 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.702419    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1bc8942-bca2-45de-9d81-31183bbf0505-must-gather-output\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.702747 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.702729    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1bc8942-bca2-45de-9d81-31183bbf0505-must-gather-output\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.711072 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.711050    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhm2k\" (UniqueName: \"kubernetes.io/projected/b1bc8942-bca2-45de-9d81-31183bbf0505-kube-api-access-lhm2k\") pod \"must-gather-7547d\" (UID: \"b1bc8942-bca2-45de-9d81-31183bbf0505\") " pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.769349 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.769306    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:38.853322 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.853290    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgqkp/must-gather-7547d"
Aug 01 00:12:38.984841 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.984814    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgqkp/must-gather-7547d"]
Aug 01 00:12:38.987034 ip-10-0-132-129 kubenswrapper[2536]: W0801 00:12:38.987007    2536 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1bc8942_bca2_45de_9d81_31183bbf0505.slice/crio-e0d83bc5e7e69066dbbc217fa1545f004adaf3fa80b913b83e9e28092e790209 WatchSource:0}: Error finding container e0d83bc5e7e69066dbbc217fa1545f004adaf3fa80b913b83e9e28092e790209: Status 404 returned error can't find the container with id e0d83bc5e7e69066dbbc217fa1545f004adaf3fa80b913b83e9e28092e790209
Aug 01 00:12:38.988864 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:38.988843    2536 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:12:39.511018 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:39.510979    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgqkp/must-gather-7547d" event={"ID":"b1bc8942-bca2-45de-9d81-31183bbf0505","Type":"ContainerStarted","Data":"e0d83bc5e7e69066dbbc217fa1545f004adaf3fa80b913b83e9e28092e790209"}
Aug 01 00:12:41.524366 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:41.524319    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgqkp/must-gather-7547d" event={"ID":"b1bc8942-bca2-45de-9d81-31183bbf0505","Type":"ContainerStarted","Data":"8dfd261973ee9a6a9df10287af760bb0a38787f091abd5bffdd8514d4cc4ec8d"}
Aug 01 00:12:41.524827 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:41.524383    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgqkp/must-gather-7547d" event={"ID":"b1bc8942-bca2-45de-9d81-31183bbf0505","Type":"ContainerStarted","Data":"4befd0c99a80ffc0457e1ee7e934406535ca86e2397621231fbea8e585f1c0d0"}
Aug 01 00:12:41.541002 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:41.540951    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xgqkp/must-gather-7547d" podStartSLOduration=2.055670002 podStartE2EDuration="3.540935377s" totalImagesPullingTime="1.485265375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:41.538718101 +0000 UTC m=+1586.503913725" watchObservedRunningTime="2026-08-01 00:12:41.540935377 +0000 UTC m=+1586.506131035"
Aug 01 00:12:41.854105 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:41.854067    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dx87q_f5475fe4-6ba2-4652-84f5-0f5a887b4a0e/global-pull-secret-syncer/0.log"
Aug 01 00:12:41.941576 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:41.941539    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log"
Aug 01 00:12:42.040132 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:42.040098    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-129.ec2.internal_8e48bbd57a2cb073c5f8cf46fee7320a/haproxy/0.log"
Aug 01 00:12:43.790307 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:43.790266    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:45.188520 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:45.188467    2536 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8crd_e59a8686-fd5f-4ab2-a476-8379aed3312a/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:46.068718 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.068629    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/alertmanager/0.log"
Aug 01 00:12:46.089830 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.089792    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/config-reloader/0.log"
Aug 01 00:12:46.111037 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.111010    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/kube-rbac-proxy-web/0.log"
Aug 01 00:12:46.133839 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.133790    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/kube-rbac-proxy/0.log"
Aug 01 00:12:46.157666 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.157635    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/kube-rbac-proxy-metric/0.log"
Aug 01 00:12:46.190382 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.190350    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/prom-label-proxy/0.log"
Aug 01 00:12:46.213389 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.213355    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e05d46a-9ef3-4434-89d3-63db02d44a79/init-config-reloader/0.log"
Aug 01 00:12:46.256509 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.256308    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-hpkqg_75c48d66-57f1-495e-97d3-8f735cf965fe/cluster-monitoring-operator/0.log"
Aug 01 00:12:46.359165 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.359084    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cb6bf7c85-94hw6_5b2546c4-d4bf-4d62-8bdc-11e3cdfe3218/metrics-server/0.log"
Aug 01 00:12:46.594328 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.594287    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2hwq_c0b25a94-caff-47c6-ab2b-063f50800445/node-exporter/0.log"
Aug 01 00:12:46.640079 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.639959    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2hwq_c0b25a94-caff-47c6-ab2b-063f50800445/kube-rbac-proxy/0.log"
Aug 01 00:12:46.662775 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.662744    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2hwq_c0b25a94-caff-47c6-ab2b-063f50800445/init-textfile/0.log"
Aug 01 00:12:46.927104 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.927012    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-swkrv_5d4ef30d-1ae1-4a04-a26d-74e6c1267baf/prometheus-operator/0.log"
Aug 01 00:12:46.948322 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.948295    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-swkrv_5d4ef30d-1ae1-4a04-a26d-74e6c1267baf/kube-rbac-proxy/0.log"
Aug 01 00:12:46.975459 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:46.975427    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-tmmsj_67fb135f-4240-445a-a77b-49fc67eca297/prometheus-operator-admission-webhook/0.log"
Aug 01 00:12:47.000726 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.000688    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cc97f9d5-wwjzg_20f75156-54e4-4161-9f30-530c4a176ad0/telemeter-client/0.log"
Aug 01 00:12:47.021786 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.021750    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cc97f9d5-wwjzg_20f75156-54e4-4161-9f30-530c4a176ad0/reload/0.log"
Aug 01 00:12:47.043758 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.043729    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cc97f9d5-wwjzg_20f75156-54e4-4161-9f30-530c4a176ad0/kube-rbac-proxy/0.log"
Aug 01 00:12:47.075175 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.075141    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/thanos-query/0.log"
Aug 01 00:12:47.095980 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.095939    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/kube-rbac-proxy-web/0.log"
Aug 01 00:12:47.115657 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.115627    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/kube-rbac-proxy/0.log"
Aug 01 00:12:47.135449 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.135414    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/prom-label-proxy/0.log"
Aug 01 00:12:47.158687 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.158661    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/kube-rbac-proxy-rules/0.log"
Aug 01 00:12:47.183650 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:47.183570    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8fff5d969-5p94j_fd92ec36-2398-44dc-b554-2a461bfb083d/kube-rbac-proxy-metrics/0.log"
Aug 01 00:12:48.432859 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:48.432825    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-zgvjl_316ac0b7-e394-405e-b5eb-3c5575a64dbe/networking-console-plugin/0.log"
Aug 01 00:12:48.958848 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:48.958820    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/1.log"
Aug 01 00:12:48.968198 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:48.968168    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zdghk_7e6b3eb5-7d66-4655-b38f-863883402a84/console-operator/2.log"
Aug 01 00:12:49.914745 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:49.914710    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-2kqrw_b87aa9ef-45e9-4918-a2e5-96188fc8f544/volume-data-source-validator/0.log"
Aug 01 00:12:49.998150 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:49.998109    2536 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"]
Aug 01 00:12:50.003938 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.003903    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.010824 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.010791    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"]
Aug 01 00:12:50.127709 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.127668    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-proc\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.127901 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.127798    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-sys\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.127901 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.127834    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-podres\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.127901 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.127884    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-lib-modules\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.128085 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.127954    2536 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb94w\" (UniqueName: \"kubernetes.io/projected/ab7a53fc-ef40-4746-816b-c197cf5c9f63-kube-api-access-gb94w\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229267 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229178    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb94w\" (UniqueName: \"kubernetes.io/projected/ab7a53fc-ef40-4746-816b-c197cf5c9f63-kube-api-access-gb94w\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229267 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229255    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-proc\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229303    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-sys\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229337    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-podres\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229384    2536 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-lib-modules\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229385    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-proc\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229468 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229449    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-sys\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229587    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-podres\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.229756 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.229598    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab7a53fc-ef40-4746-816b-c197cf5c9f63-lib-modules\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.237566 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.237539    2536 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb94w\" (UniqueName: \"kubernetes.io/projected/ab7a53fc-ef40-4746-816b-c197cf5c9f63-kube-api-access-gb94w\") pod \"perf-node-gather-daemonset-9xdnb\" (UID: \"ab7a53fc-ef40-4746-816b-c197cf5c9f63\") " pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.322891 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.322641    2536 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:50.498054 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.497547    2536 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"]
Aug 01 00:12:50.585780 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.585746    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb" event={"ID":"ab7a53fc-ef40-4746-816b-c197cf5c9f63","Type":"ContainerStarted","Data":"3a598e28dccfbf369890dd9dd48b1e00c15a4df50a07c2c4e4190e9f6b4714d5"}
Aug 01 00:12:50.820372 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.820330    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgcbc_d563d196-7d23-4c7c-adc7-4f1c8b10f1c7/dns/0.log"
Aug 01 00:12:50.840537 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.840477    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgcbc_d563d196-7d23-4c7c-adc7-4f1c8b10f1c7/kube-rbac-proxy/0.log"
Aug 01 00:12:50.994245 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:50.994220    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zlm46_dfcb504d-a872-4bfc-ad0c-976806a7def6/dns-node-resolver/0.log"
Aug 01 00:12:51.474692 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:51.474665    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmzcw_40464426-09cc-4746-93bc-2ec417a8b4c2/node-ca/0.log"
Aug 01 00:12:51.595432 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:51.595395    2536 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb" event={"ID":"ab7a53fc-ef40-4746-816b-c197cf5c9f63","Type":"ContainerStarted","Data":"ad329c195b696cf068977296ab2061007c7b478e079447b98f06855c58b67390"}
Aug 01 00:12:51.596362 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:51.596337    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:51.611477 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:51.611427    2536 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb" podStartSLOduration=2.6114119369999997 podStartE2EDuration="2.611411937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:51.610829826 +0000 UTC m=+1596.576025449" watchObservedRunningTime="2026-08-01 00:12:51.611411937 +0000 UTC m=+1596.576607560"
Aug 01 00:12:52.236990 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:52.236947    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4768f676-zrpn9_cbfa1dd6-1203-4c70-8e8c-48b5d95814a0/router/0.log"
Aug 01 00:12:52.568815 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:52.568785    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr4vj_ad02ce1a-ff7b-4c13-9c1c-04aa47ddf150/serve-healthcheck-canary/0.log"
Aug 01 00:12:52.912111 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:52.912041    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-g2zqs_d7fd0abc-48ff-4224-8cf3-2bb4b369193a/insights-operator/1.log"
Aug 01 00:12:52.912111 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:52.912077    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-g2zqs_d7fd0abc-48ff-4224-8cf3-2bb4b369193a/insights-operator/0.log"
Aug 01 00:12:53.051972 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:53.051941    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxz7r_2de10c36-1b46-406c-91e3-5ca95e0d0d7a/kube-rbac-proxy/0.log"
Aug 01 00:12:53.069233 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:53.069205    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxz7r_2de10c36-1b46-406c-91e3-5ca95e0d0d7a/exporter/0.log"
Aug 01 00:12:53.088597 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:53.088566    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dxz7r_2de10c36-1b46-406c-91e3-5ca95e0d0d7a/extractor/0.log"
Aug 01 00:12:55.268704 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:55.268673    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-674d959fb8-r8hst_14430194-c798-4faa-a171-f4d4b8515152/manager/0.log"
Aug 01 00:12:55.452818 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:55.452783    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-27kk2_fca21df4-4003-4197-baaa-188e448a3e32/manager/0.log"
Aug 01 00:12:55.522714 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:55.522644    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-mdgp4_dff8f2aa-2b39-45f9-b725-0ab1e36f443e/seaweedfs/0.log"
Aug 01 00:12:58.619170 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:58.619136    2536 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xgqkp/perf-node-gather-daemonset-9xdnb"
Aug 01 00:12:59.419653 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:59.419618    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-vjzn8_2ad66796-3dd4-48e5-856a-d7deaf771563/kube-storage-version-migrator-operator/1.log"
Aug 01 00:12:59.420388 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:12:59.420355    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-vjzn8_2ad66796-3dd4-48e5-856a-d7deaf771563/kube-storage-version-migrator-operator/0.log"
Aug 01 00:13:00.498527 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.498440    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:13:00.519134 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.519101    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/egress-router-binary-copy/0.log"
Aug 01 00:13:00.541427 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.541402    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/cni-plugins/0.log"
Aug 01 00:13:00.566704 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.566679    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/bond-cni-plugin/0.log"
Aug 01 00:13:00.589828 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.589802    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/routeoverride-cni/0.log"
Aug 01 00:13:00.610407 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.610371    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/whereabouts-cni-bincopy/0.log"
Aug 01 00:13:00.631305 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:00.631261    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n49r5_241aa0ea-8d3f-4d63-b482-12fb581f1810/whereabouts-cni/0.log"
Aug 01 00:13:01.009668 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:01.009573    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lv99k_4d510acf-e79e-4e38-8faa-58281e5def06/kube-multus/0.log"
Aug 01 00:13:01.136460 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:01.136432    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq5g7_d0eac260-1645-44ba-9ed6-549540cdef6d/network-metrics-daemon/0.log"
Aug 01 00:13:01.154018 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:01.153994    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq5g7_d0eac260-1645-44ba-9ed6-549540cdef6d/kube-rbac-proxy/0.log"
Aug 01 00:13:02.051946 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.051908    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/ovn-controller/0.log"
Aug 01 00:13:02.095316 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.095286    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/ovn-acl-logging/0.log"
Aug 01 00:13:02.137594 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.137563    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/kube-rbac-proxy-node/0.log"
Aug 01 00:13:02.174158 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.174134    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:13:02.207777 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.207748    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/northd/0.log"
Aug 01 00:13:02.243913 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.243884    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/nbdb/0.log"
Aug 01 00:13:02.283939 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.283915    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/sbdb/0.log"
Aug 01 00:13:02.338821 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:02.338736    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d9cnv_e2d84a4a-4b9e-4903-8d41-f998c63622ee/ovnkube-controller/0.log"
Aug 01 00:13:03.534440 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:03.534401    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-65bdbbff45-5dmjq_134b2375-9a90-46ad-9980-5c5467e1a381/check-endpoints/0.log"
Aug 01 00:13:03.555995 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:03.555970    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8kmsg_0fc4f897-6876-4f7e-a1d5-f897546761fb/network-check-target-container/0.log"
Aug 01 00:13:04.566528 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:04.566499    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-x9kdv_8ff29c5e-04e1-4d86-85e8-85a25196a97b/iptables-alerter/0.log"
Aug 01 00:13:05.173891 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:05.173858    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6fnf7_d6d3af55-3144-42d1-98f9-8b11c16bb667/tuned/0.log"
Aug 01 00:13:06.854034 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:06.854003    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-559cbf5745-cs2kt_a39d93a8-1777-4329-ab31-24a680418b44/cluster-samples-operator/0.log"
Aug 01 00:13:06.868205 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:06.868179    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-559cbf5745-cs2kt_a39d93a8-1777-4329-ab31-24a680418b44/cluster-samples-operator-watch/0.log"
Aug 01 00:13:07.787913 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:07.787824    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-784f97bcf6-wvjfj_1fe52f27-2747-45aa-a384-14548caf51b6/service-ca-operator/1.log"
Aug 01 00:13:07.789121 ip-10-0-132-129 kubenswrapper[2536]: I0801 00:13:07.789094    2536 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-784f97bcf6-wvjfj_1fe52f27-2747-45aa-a384-14548caf51b6/service-ca-operator/0.log"