Jul 01 20:44:13.092347 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 20:44:13.092359 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 20:44:13.092366 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 20:44:13.092578 ip-10-0-129-137 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 20:44:23.433923 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 20:44:23.433937 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2f3300555cb64b59ae39c52be473a2f0 --
Jul 01 20:46:06.210207 ip-10-0-129-137 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.550328    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554771    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554782    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554786    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554789    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554792    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554795    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554797    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.657768 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554800    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554802    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554805    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554808    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554810    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554813    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554817    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554821    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554824    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554827    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554830    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554832    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554835    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554837    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554840    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554843    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554845    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554848    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554850    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554854    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.659388 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554857    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554862    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554865    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554868    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554871    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554874    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554877    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554879    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554883    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554886    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554888    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554891    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554894    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554896    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554900    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554902    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554905    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554907    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554910    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554912    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.661847 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554915    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554918    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554920    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554923    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554925    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554927    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554944    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554949    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554956    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554961    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554966    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554969    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554971    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554975    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554977    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554980    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554982    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554985    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554988    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.662641 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554990    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554993    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554996    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.554998    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555001    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555004    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555006    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555009    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555011    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555014    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555017    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555020    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555022    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555025    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555028    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555031    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555034    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555037    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555039    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555042    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.663894 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555044    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555048    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555052    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555055    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555057    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555060    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555063    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555065    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555068    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555071    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555073    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555076    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555078    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555777    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555787    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555792    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555795    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555798    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555801    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555804    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.672636 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555807    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555809    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555812    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555815    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555817    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555820    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555823    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555826    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555829    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555831    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555835    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555837    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555839    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555842    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555845    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555848    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555850    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555852    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555855    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555857    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.673863 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555860    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555862    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555865    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555867    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555871    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555873    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555876    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555878    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555881    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555883    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555886    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555889    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555891    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555894    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555898    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555900    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555902    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555905    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555907    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555911    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.674911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555913    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555916    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555919    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555921    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555924    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555926    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555929    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555948    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555951    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555954    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555957    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555961    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555965    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555968    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555971    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555974    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555977    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555980    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555983    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.676798 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555985    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555988    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555991    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555993    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555996    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.555998    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556001    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556003    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556005    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556008    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556010    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556013    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556016    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556019    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556022    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556024    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556028    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556032    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556035    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.677651 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556038    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556041    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556043    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556046    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556048    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556051    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556053    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556056    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556058    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556061    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556063    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556066    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556068    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556071    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556414    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556418    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556421    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556425    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556428    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556432    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.678456 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556435    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556438    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556440    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556443    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556445    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556448    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556454    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556457    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556460    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556462    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556465    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556468    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556470    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556475    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556479    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556482    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556485    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556488    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556491    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556494    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.679761 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556497    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.680398 ip-10-0-129-137 systemd[1]: Started Kubernetes Kubelet.
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556500    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556502    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556505    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556507    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556510    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556513    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556515    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556518    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556521    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556523    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556526    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556528    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556531    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556533    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556536    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556538    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556541    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556543    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.680905 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556546    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556550    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556552    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556555    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556558    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556561    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556564    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556567    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556569    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556572    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556574    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556579    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556582    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556585    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556587    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556590    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556593    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556596    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556598    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.681919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556601    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556603    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556606    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556608    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556611    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556614    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556619    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556622    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556624    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556627    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556629    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556632    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556635    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556637    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556640    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556645    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556647    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556650    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556652    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556655    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.682799 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556657    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556660    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556663    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556665    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556668    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556670    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556673    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556675    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556678    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556681    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556683    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556685    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556688    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556690    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.556693    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556759    2554 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556766    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556772    2554 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556778    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556782    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556785    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 20:46:06.684024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556789    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556795    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556798    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556801    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556804    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556807    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556813    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556816    2554 flags.go:64] FLAG: --cgroup-root=""
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556819    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556822    2554 flags.go:64] FLAG: --client-ca-file=""
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556825    2554 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556828    2554 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556831    2554 flags.go:64] FLAG: --cluster-domain=""
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556834    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556837    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556840    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556843    2554 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556847    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556850    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556853    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556856    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556859    2554 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556862    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556864    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 20:46:06.685129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556868    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556870    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556874    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556877    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556880    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556884    2554 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556887    2554 flags.go:64] FLAG: --enable-server="true"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556890    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556894    2554 flags.go:64] FLAG: --event-burst="100"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556897    2554 flags.go:64] FLAG: --event-qps="50"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556900    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556903    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556906    2554 flags.go:64] FLAG: --eviction-hard=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556909    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556913    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556922    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556925    2554 flags.go:64] FLAG: --eviction-soft=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556928    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556943    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556947    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556949    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556952    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556955    2554 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556958    2554 flags.go:64] FLAG: --feature-gates=""
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556961    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 20:46:06.686120 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556964    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556967    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556970    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556973    2554 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556976    2554 flags.go:64] FLAG: --help="false"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556979    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556982    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556985    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556988    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556991    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556994    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.556997    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557002    2554 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557005    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557007    2554 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557010    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557013    2554 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557016    2554 flags.go:64] FLAG: --kube-reserved=""
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557018    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557021    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557024    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557029    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557031    2554 flags.go:64] FLAG: --lock-file=""
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557042    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 20:46:06.686958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557045    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557048    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557053    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557055    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557058    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557061    2554 flags.go:64] FLAG: --logging-format="text"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557064    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557067    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557070    2554 flags.go:64] FLAG: --manifest-url=""
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557072    2554 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557076    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557079    2554 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557083    2554 flags.go:64] FLAG: --max-pods="110"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557086    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557088    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557091    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557094    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557097    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557100    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557102    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557111    2554 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557114    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557117    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 20:46:06.687635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557120    2554 flags.go:64] FLAG: --pod-cidr=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557123    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557128    2554 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557131    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557134    2554 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557137    2554 flags.go:64] FLAG: --port="10250"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557139    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557143    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e317ad53f5cb30ad"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557146    2554 flags.go:64] FLAG: --qos-reserved=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557154    2554 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557157    2554 flags.go:64] FLAG: --register-node="true"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557160    2554 flags.go:64] FLAG: --register-with-taints=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557164    2554 flags.go:64] FLAG: --registry-burst="10"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557166    2554 flags.go:64] FLAG: --registry-qps="5"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557169    2554 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557172    2554 flags.go:64] FLAG: --reserved-memory=""
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557176    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557179    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557182    2554 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557184    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557187    2554 flags.go:64] FLAG: --runonce="false"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557190    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557193    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557196    2554 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557199    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 20:46:06.688469 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557202    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557204    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557207    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557210    2554 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557214    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557217    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557220    2554 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557222    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557225    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557228    2554 flags.go:64] FLAG: --system-cgroups=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557231    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557236    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557239    2554 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557242    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557246    2554 flags.go:64] FLAG: --tls-min-version=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557249    2554 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557252    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557258    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557261    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557264    2554 flags.go:64] FLAG: --v="2"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557267    2554 flags.go:64] FLAG: --version="false"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557271    2554 flags.go:64] FLAG: --vmodule=""
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557274    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557277    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557382    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.689269 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557386    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557389    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557392    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557395    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557398    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557400    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557403    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557405    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557408    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557411    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557413    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557417    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557419    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557422    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557424    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557427    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557429    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557432    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557435    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.690524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557437    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557441    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557446    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557449    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557451    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557454    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557461    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557463    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557466    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557468    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557471    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557474    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557477    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557479    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557482    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557484    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557486    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557489    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557491    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557494    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.691600 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557496    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557499    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557501    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557504    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557507    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557510    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557512    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557515    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557517    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557520    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557522    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557525    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557528    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557530    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557534    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557537    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557539    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557542    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557545    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557553    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.692415 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557555    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557558    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557560    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557563    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557566    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557568    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557571    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557573    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557576    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557578    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557580    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557583    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557585    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557588    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557590    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557593    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557597    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557599    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557602    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557604    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.693029 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557607    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557609    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557612    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557614    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557617    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557619    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557622    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557625    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557627    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557630    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557632    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557635    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557643    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557645    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557648    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557650    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557654    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557658    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.693689 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.557661    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.557667    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.564393    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.564406    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.564425    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.564431    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564483    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564487    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564491    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564493    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564496    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564499    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564502    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564505    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.695408 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564507    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564509    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564512    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564515    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564518    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564520    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564523    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564525    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564528    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564530    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564533    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564536    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564542    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564545    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564547    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564550    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564553    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564555    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564557    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564560    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.696246 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564563    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564566    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564568    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564573    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564577    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564580    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564582    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564585    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564588    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564591    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564593    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564596    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564600    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564603    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564606    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564608    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564610    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564613    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564616    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.697979 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564619    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564623    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564626    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564629    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564631    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564635    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564638    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564640    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564642    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564645    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564648    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564650    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564653    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564655    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564658    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564661    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564664    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564666    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564668    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564671    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.698907 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564673    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564676    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564679    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564683    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564686    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564690    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564693    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564696    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564698    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564701    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564703    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564706    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564709    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564711    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564714    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564717    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564719    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564723    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564726    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564728    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.700044 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564731    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564733    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564736    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564738    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564741    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564743    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564746    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564748    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564751    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564754    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564756    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564759    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.564764    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564868    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564872    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:06.701118 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564875    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564878    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564881    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564885    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564888    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564890    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564893    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564895    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564898    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564901    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564904    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564907    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564909    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564912    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564915    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564918    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564920    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564923    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564926    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564928    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:06.701911 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564946    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564950    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564952    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564955    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564957    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564960    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564963    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564966    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564968    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564971    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564973    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564976    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564978    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564981    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564983    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564989    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564991    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564994    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564996    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:06.702852 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.564999    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565001    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565004    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565006    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565009    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565011    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565014    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565018    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565021    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565024    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565026    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565029    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565031    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565034    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565036    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565039    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565041    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565043    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565046    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565048    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:06.703609 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565052    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565055    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565059    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565062    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565065    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565068    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565071    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565074    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565078    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565081    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565084    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565086    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565089    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565091    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565094    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565096    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565099    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565102    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565104    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565109    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:06.815623 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565111    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565114    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565116    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565119    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565121    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565124    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565126    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565129    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565131    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565134    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565136    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565139    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565141    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565144    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565148    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565151    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565154    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:06.836697 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:06.565157    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.565162    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.565250    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.629538    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.630982    2554 server.go:1019] "Starting client certificate rotation"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.631082    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.631112    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.633524    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.637537    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.648656    2554 log.go:25] "Validated CRI v1 runtime API"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.650826    2554 log.go:25] "Validated CRI v1 image API"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.651705    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 20:46:06.837545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.653848    2554 fs.go:135] Filesystem UUIDs: map[1c143521-5525-47de-be28-03944a6fc8a5:/dev/nvme0n1p4 23731226-8b96-4d71-8c2d-41204cda826e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 01 20:46:06.838364 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.653867    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.659406    2554 manager.go:216] Machine: {Timestamp:2026-07-01 20:46:06.65873016 +0000 UTC m=+0.342214057 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100291 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f99b7a65fdd18bae1e841d444cd52 SystemUUID:ec2f99b7-a65f-dd18-bae1-e841d444cd52 BootID:2f330055-5cb6-4b59-ae39-c52be473a2f0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ea:db:12:dd:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ea:db:12:dd:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:2a:5a:0f:66:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.659536    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.659623    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.662792    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.662822    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.663879    2554 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.663896    2554 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 20:46:06.839990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.663907    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.664684    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.665540    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.666523    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.670815    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.670994    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671028    2554 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671041    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671055    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671066    2554 kubelet.go:398] "Adding apiserver pod source"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671076    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671451    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.671468    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.673482    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.674748    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.674775    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675371    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675390    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675399    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675407    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675416    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675421    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675428    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675434    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675440    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675446    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675455    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.675460    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.676393    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.676401    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.679428    2554 server.go:1268] "Started kubelet"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.679709    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.679694    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.679764    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.681512    2554 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.682353    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.684797    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.684805    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.684808    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 20:46:06.840392 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.685527    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.685634    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18be455ca432f015  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-01 20:46:06.679404565 +0000 UTC m=+0.362888459,LastTimestamp:2026-07-01 20:46:06.679404565 +0000 UTC m=+0.362888459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689017    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689039    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.689169    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689593    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689623    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689640    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689716    2554 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689724    2554 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689809    2554 factory.go:55] Registering systemd factory
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.689823    2554 factory.go:223] Registration of the systemd container factory successfully
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690060    2554 factory.go:153] Registering CRI-O factory
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690075    2554 factory.go:223] Registration of the crio container factory successfully
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690123    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690146    2554 factory.go:103] Registering Raw factory
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690160    2554 manager.go:1196] Started watching for new ooms in manager
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.690708    2554 manager.go:349] Starting recovery of all containers
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.698247    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.698315    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.700106    2554 manager.go:354] Recovery completed
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.704855    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.708619    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.708646    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.708659    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709137    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709148    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709169    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709190    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709780    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709809    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.709818    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.711323    2554 policy_none.go:49] "None policy: Start"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.711341    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 20:46:06.842824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.711351    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.713269    2554 policy_none.go:47] "Start"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.720905    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18be455ca5f0f077  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-01 20:46:06.708633719 +0000 UTC m=+0.392117634,LastTimestamp:2026-07-01 20:46:06.708633719 +0000 UTC m=+0.392117634,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.723243    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dg2f8"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.729318    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18be455ca5f1345d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-01 20:46:06.708651101 +0000 UTC m=+0.392134997,LastTimestamp:2026-07-01 20:46:06.708651101 +0000 UTC m=+0.392134997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.730648    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dg2f8"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.763545    2554 manager.go:337] "Starting Device Plugin manager"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.763573    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.763582    2554 server.go:87] "Starting device plugin registration server"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.763812    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.763823    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.763943    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.764013    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.764021    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.764552    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.764618    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.765468    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.766632    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.766669    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.766687    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:06.766720    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 20:46:06.844056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.769038    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:06.962109 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:06.864789    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.597616 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597592    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.597748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597632    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.597748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597647    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.597748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597674    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.597748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597559    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"]
Jul 01 20:46:07.597748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.597746    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.599174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.599159    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.599240 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.599186    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.599240 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.599195    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.601494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.601484    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.601610 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.601597    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.601660 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.601624    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.602145 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602130    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.602227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602153    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.602227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602151    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.602227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602164    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.602227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602179    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.602227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.602194    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.603461 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.603443    2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.604391 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.604377    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.604468 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.604405    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.604994 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.604980    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.605057 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.605008    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.605057 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.605018    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.606672 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.606659    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.606737 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.606675    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-137.ec2.internal\": node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:07.630171 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.630159    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 20:46:07.630310 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.630294    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 20:46:07.630369 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.630325    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{}\" for node \"ip-10-0-129-137.ec2.internal\": Patch \"https://a75a15c1d97fd42bfbf9a7f1ec60ad48-29c253c988586678.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-129-137.ec2.internal/status?timeout=10s\": read tcp 10.0.129.137:55288->100.29.69.187:6443: use of closed network connection"
Jul 01 20:46:07.630369 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.630279    2554 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a75a15c1d97fd42bfbf9a7f1ec60ad48-29c253c988586678.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-137.ec2.internal.18be455ca5f1345d\": read tcp 10.0.129.137:55288->100.29.69.187:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18be455ca5f1345d  default   5927 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-01 20:46:06 +0000 UTC,LastTimestamp:2026-07-01 20:46:07.599190587 +0000 UTC m=+1.282674480,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}"
Jul 01 20:46:07.636471 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.636456    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.640536 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.640524    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.658788 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.658773    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:07.697730 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.697706    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.697730 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.697726    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.697842 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.697742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d191265e6bbab3f8d2d9679a92fcb37-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"0d191265e6bbab3f8d2d9679a92fcb37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.736803 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.736777    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 20:41:06 +0000 UTC" deadline="2028-03-19 18:30:10.110848062 +0000 UTC"
Jul 01 20:46:07.736803 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.736800    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15045h44m2.374050413s"
Jul 01 20:46:07.756926 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.756895    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.757776 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.757754    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.757840 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.757780    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.757840 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.757789    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.758846 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.758833    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:07.798810 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.798884 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798816    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.798884 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798830    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d191265e6bbab3f8d2d9679a92fcb37-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"0d191265e6bbab3f8d2d9679a92fcb37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.798884 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798862    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d191265e6bbab3f8d2d9679a92fcb37-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"0d191265e6bbab3f8d2d9679a92fcb37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.799001 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798891    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.799001 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.798891    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c21388212b058e3b39cb3124c51bb32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"7c21388212b058e3b39cb3124c51bb32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.829981 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.829923    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:07.859100 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.859074    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:07.939478 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.939453    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.943489 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.943444    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:07.959421 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:07.959402    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Jul 01 20:46:07.980838 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.980819    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:07.989725 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:07.989709    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:08.079638 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.079620    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:08.080436 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.080404    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Jul 01 20:46:08.089227 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.089212    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:08.102968 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.102952    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:08.476647 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:08.476619    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c21388212b058e3b39cb3124c51bb32.slice/crio-a8dc2e6f2b5a2dbeb470510a3ecb9d5d14358574cd9d2e18d439d0f4d0507838 WatchSource:0}: Error finding container a8dc2e6f2b5a2dbeb470510a3ecb9d5d14358574cd9d2e18d439d0f4d0507838: Status 404 returned error can't find the container with id a8dc2e6f2b5a2dbeb470510a3ecb9d5d14358574cd9d2e18d439d0f4d0507838
Jul 01 20:46:08.476914 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:08.476894    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d191265e6bbab3f8d2d9679a92fcb37.slice/crio-f38b26371dedcd4c891658553f9c0aecb417884432196db50632b521238831ca WatchSource:0}: Error finding container f38b26371dedcd4c891658553f9c0aecb417884432196db50632b521238831ca: Status 404 returned error can't find the container with id f38b26371dedcd4c891658553f9c0aecb417884432196db50632b521238831ca
Jul 01 20:46:08.482188 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.482175    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:46:08.671960 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.671926    2554 apiserver.go:52] "Watching apiserver"
Jul 01 20:46:08.681064 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.681039    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 20:46:08.681368 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.681348    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kdd8w","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qbsrl","openshift-dns/node-resolver-7qn9x","openshift-image-registry/node-ca-24krb","openshift-multus/multus-additional-cni-plugins-chk96","openshift-multus/multus-q7xh8","openshift-multus/network-metrics-daemon-7fwz5","kube-system/global-pull-secret-syncer-cbm9x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","openshift-network-diagnostics/network-check-target-9scdv","openshift-network-operator/iptables-alerter-wqq4l","openshift-ovn-kubernetes/ovnkube-node-4m68j"]
Jul 01 20:46:08.684769 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.684754    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.686690 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.686672    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.687329 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.687312    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 20:46:08.687416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.687317    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 20:46:08.687416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.687317    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cndv8\""
Jul 01 20:46:08.688610 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.688596    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.688739 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.688718    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7fxsd\""
Jul 01 20:46:08.688876 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.688862    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.688986 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.688972    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.689561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.689547    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:08.690538 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.690499    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.690841 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.690813    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dqdtj\""
Jul 01 20:46:08.690841 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.690824    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.690975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.690892    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.692474 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.692458    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.692682 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.692572    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 20:46:08.693029 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.693012    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.693086 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.693031    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.693209 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.693198    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cqnrf\""
Jul 01 20:46:08.694415 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.694400    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.695339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695289    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gzkl\""
Jul 01 20:46:08.695339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695301    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 20:46:08.695339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695308    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.695339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695317    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.695339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695327    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 20:46:08.695573 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.695343    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 20:46:08.696501 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.696488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.696606 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.696550    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:08.696714 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.696702    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 20:46:08.696774 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.696739    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-47stp\""
Jul 01 20:46:08.698520 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.698503    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.698577 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.698564    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:08.698793 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.698780    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:08.700618 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.700604    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.701978 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.701914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysconfig\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.701978 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.701956    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-lib-modules\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.701978 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.701977    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-tuned\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.701997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-tmp\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702018    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twps5\" (UniqueName: \"kubernetes.io/projected/153889c1-36ed-424f-ad2e-4e3d98b84921-kube-api-access-twps5\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.702091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702034    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-kubernetes\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-conf\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-run\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702093    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-var-lib-kubelet\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702106    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frhx\" (UniqueName: \"kubernetes.io/projected/add06c46-3ee3-4b0b-b239-9b2356dd099e-kube-api-access-9frhx\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-tmp-dir\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljkx\" (UniqueName: \"kubernetes.io/projected/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-kube-api-access-qljkx\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153889c1-36ed-424f-ad2e-4e3d98b84921-serviceca\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-host\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702193    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-konnectivity-ca\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.702219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-modprobe-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702228    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702240    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-systemd\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-sys\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702291    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-hosts-file\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702306    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153889c1-36ed-424f-ad2e-4e3d98b84921-host\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.702446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-agent-certs\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.702722 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702625    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:08.702722 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.702701    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:08.702827 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702793    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 20:46:08.702827 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702824    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.702997 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.702983    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74g5g\""
Jul 01 20:46:08.703066 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.703052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.704645 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.704628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.706791 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.706777    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 20:46:08.706883 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.706821    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.707013 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.706995    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.707098 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.707054    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lqdtg\""
Jul 01 20:46:08.707158 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.707108    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.710049 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.710034    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 20:46:08.710140 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.710053    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 20:46:08.711149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.711138    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 20:46:08.711213 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.711168    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 20:46:08.713703 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.711649    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 20:46:08.713703 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.711726    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 20:46:08.713703 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.711906    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rx4qq\""
Jul 01 20:46:08.724450 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.724433    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fljmk"
Jul 01 20:46:08.732452 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.732432    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fljmk"
Jul 01 20:46:08.770485 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.770451    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"0d191265e6bbab3f8d2d9679a92fcb37","Type":"ContainerStarted","Data":"f38b26371dedcd4c891658553f9c0aecb417884432196db50632b521238831ca"}
Jul 01 20:46:08.771349 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.771333    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"7c21388212b058e3b39cb3124c51bb32","Type":"ContainerStarted","Data":"a8dc2e6f2b5a2dbeb470510a3ecb9d5d14358574cd9d2e18d439d0f4d0507838"}
Jul 01 20:46:08.790279 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.790263    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 20:46:08.803422 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.803487 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803444    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.803487 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756kb\" (UniqueName: \"kubernetes.io/projected/0c6c57df-4530-4a13-9437-4256b6ccff0d-kube-api-access-756kb\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.803487 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803501    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153889c1-36ed-424f-ad2e-4e3d98b84921-host\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803523    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysconfig\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803537    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153889c1-36ed-424f-ad2e-4e3d98b84921-host\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-lib-modules\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803562    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysconfig\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803604    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-ovn\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.803635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803631    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-lib-modules\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-host\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803667    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-bin\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803688    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwv8\" (UniqueName: \"kubernetes.io/projected/9608f8a5-7473-4876-a9de-fd6b06753e13-kube-api-access-6bwv8\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-system-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803744    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-host\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803766    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803790    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-k8s-cni-cncf-io\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-netns\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803837    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-bin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803861    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-systemd\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803882    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z597\" (UniqueName: \"kubernetes.io/projected/65b00689-c10f-4b94-a41e-292d4ed16225-kube-api-access-5z597\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803912    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-multus\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803952    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-etc-kubernetes\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.804040 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803960    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-systemd\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.803976    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-tmp\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frhx\" (UniqueName: \"kubernetes.io/projected/add06c46-3ee3-4b0b-b239-9b2356dd099e-kube-api-access-9frhx\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804025    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b00689-c10f-4b94-a41e-292d4ed16225-iptables-alerter-script\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-os-release\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804185    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-dbus\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804237    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 20:46:08.804552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804269    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-systemd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804308    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-netd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-config\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljkx\" (UniqueName: \"kubernetes.io/projected/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-kube-api-access-qljkx\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153889c1-36ed-424f-ad2e-4e3d98b84921-serviceca\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804373    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-cnibin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-socket-dir-parent\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-hostroot\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-daemon-config\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-multus-certs\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpcs\" (UniqueName: \"kubernetes.io/projected/bcfc626c-0f15-49dd-85e8-4a65410fee34-kube-api-access-qwpcs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804543    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-konnectivity-ca\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-sys\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804641    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-system-cni-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.805060 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804647    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-sys\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-var-lib-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-etc-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804700    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-script-lib\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-os-release\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-hosts-file\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804726    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/153889c1-36ed-424f-ad2e-4e3d98b84921-serviceca\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804741    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-agent-certs\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804764    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-tuned\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804776    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-hosts-file\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-env-overrides\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rw8q\" (UniqueName: \"kubernetes.io/projected/95595c7a-7522-4735-9d27-2ac1237c8f03-kube-api-access-4rw8q\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twps5\" (UniqueName: \"kubernetes.io/projected/153889c1-36ed-424f-ad2e-4e3d98b84921-kube-api-access-twps5\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-kubernetes\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.805676 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804899    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-var-lib-kubelet\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804922    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-cnibin\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-tmp-dir\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804985    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804987    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-var-lib-kubelet\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.804996    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-konnectivity-ca\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b00689-c10f-4b94-a41e-292d4ed16225-host-slash\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-systemd-units\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-netns\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-cni-binary-copy\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805160    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-modprobe-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-kubelet\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805239    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-tmp-dir\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.806131 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2mv\" (UniqueName: \"kubernetes.io/projected/8e9a83a3-4849-4baf-a9a7-781bf7974983-kube-api-access-5k2mv\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805284    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-slash\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805309    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-log-socket\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805329    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-modprobe-d\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805354    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-kubernetes\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-conf-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-device-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-conf\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805488    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-run\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805502    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-kubelet-config\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-node-log\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805529    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-kubelet\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805580    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.806529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805590    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-sysctl-conf\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806896 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805612    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.806896 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805560    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/add06c46-3ee3-4b0b-b239-9b2356dd099e-run\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.806896 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9608f8a5-7473-4876-a9de-fd6b06753e13-ovn-node-metrics-cert\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.806896 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.805653    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.807392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.807371    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-etc-tuned\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.807453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.807433    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add06c46-3ee3-4b0b-b239-9b2356dd099e-tmp\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.807632 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.807617    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b26c1158-7a44-4ec3-afc7-f3b0c2c4a047-agent-certs\") pod \"konnectivity-agent-kdd8w\" (UID: \"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047\") " pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.815745 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.815727    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frhx\" (UniqueName: \"kubernetes.io/projected/add06c46-3ee3-4b0b-b239-9b2356dd099e-kube-api-access-9frhx\") pod \"tuned-qbsrl\" (UID: \"add06c46-3ee3-4b0b-b239-9b2356dd099e\") " pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:08.815926 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.815901    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twps5\" (UniqueName: \"kubernetes.io/projected/153889c1-36ed-424f-ad2e-4e3d98b84921-kube-api-access-twps5\") pod \"node-ca-24krb\" (UID: \"153889c1-36ed-424f-ad2e-4e3d98b84921\") " pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:08.815987 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.815977    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljkx\" (UniqueName: \"kubernetes.io/projected/78922f2f-7ca3-468e-ae75-5a9cb7ca1563-kube-api-access-qljkx\") pod \"node-resolver-7qn9x\" (UID: \"78922f2f-7ca3-468e-ae75-5a9cb7ca1563\") " pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:08.872112 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.872092    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:08.906452 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906431    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-system-cni-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.906452 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906453    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-var-lib-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-etc-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-script-lib\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906494    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-os-release\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906524    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-env-overrides\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906529    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-var-lib-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906546    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-etc-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rw8q\" (UniqueName: \"kubernetes.io/projected/95595c7a-7522-4735-9d27-2ac1237c8f03-kube-api-access-4rw8q\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.906588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-cnibin\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906595    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906599    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-os-release\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b00689-c10f-4b94-a41e-292d4ed16225-host-slash\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906642    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-system-cni-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906658    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-systemd-units\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-netns\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906687    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b00689-c10f-4b94-a41e-292d4ed16225-host-slash\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906655    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-cnibin\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-cni-binary-copy\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906723    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906729    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-systemd-units\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906751    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-kubelet\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906750    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-netns\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.906975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906776    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2mv\" (UniqueName: \"kubernetes.io/projected/8e9a83a3-4849-4baf-a9a7-781bf7974983-kube-api-access-5k2mv\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906808    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-slash\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906831    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-log-socket\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906856    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-conf-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906877    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-device-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906971    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-log-socket\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.906905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-kubelet-config\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907373    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-node-log\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907381    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-conf-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907395    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-env-overrides\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-kubelet\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907428    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-slash\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907434    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907464    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.907517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907483    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907493    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907332    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-script-lib\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9608f8a5-7473-4876-a9de-fd6b06753e13-ovn-node-metrics-cert\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907563    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-kubelet\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907582    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-device-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907607    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-node-log\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756kb\" (UniqueName: \"kubernetes.io/projected/0c6c57df-4530-4a13-9437-4256b6ccff0d-kube-api-access-756kb\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907624    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907649    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-cni-binary-copy\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907706    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907741    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907759    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-kubelet\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.908167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907842    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-kubelet-config\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907962    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.907969    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.907965    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908000    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-ovn\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908023    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-openvswitch\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.908052    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:09.408026585 +0000 UTC m=+3.091510478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908072    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-ovn\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-bin\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908115    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-run-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwv8\" (UniqueName: \"kubernetes.io/projected/9608f8a5-7473-4876-a9de-fd6b06753e13-kube-api-access-6bwv8\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908140    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-bin\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.908770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908161    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-system-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908184    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-k8s-cni-cncf-io\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908240    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-netns\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908252    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-system-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-bin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z597\" (UniqueName: \"kubernetes.io/projected/65b00689-c10f-4b94-a41e-292d4ed16225-kube-api-access-5z597\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908316    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-bin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908339    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-multus\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-etc-kubernetes\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908437    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-cni-dir\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b00689-c10f-4b94-a41e-292d4ed16225-iptables-alerter-script\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.909416 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908509    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908536    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-os-release\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908561    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-netns\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908589    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908612    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-k8s-cni-cncf-io\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908618    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-dbus\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908662    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-var-lib-cni-multus\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-systemd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-netd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908704    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-etc-kubernetes\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908719    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-config\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-cnibin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908774    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-socket-dir-parent\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910070 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908800    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-hostroot\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908822    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-daemon-config\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908859    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-multus-certs\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908886    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpcs\" (UniqueName: \"kubernetes.io/projected/bcfc626c-0f15-49dd-85e8-4a65410fee34-kube-api-access-qwpcs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.908916    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b00689-c10f-4b94-a41e-292d4ed16225-iptables-alerter-script\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909068    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-run-systemd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909117    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-cni-netd\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909124    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-dbus\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909162    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c6c57df-4530-4a13-9437-4256b6ccff0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.909191    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909252    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-hostroot\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909269    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9608f8a5-7473-4876-a9de-fd6b06753e13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909369    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909501    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcfc626c-0f15-49dd-85e8-4a65410fee34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.910708 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909541    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-daemon-config\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.909577    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:09.409541813 +0000 UTC m=+3.093025706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909617    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c6c57df-4530-4a13-9437-4256b6ccff0d-os-release\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909637    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-host-run-multus-certs\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909670    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-multus-socket-dir-parent\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.909690    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95595c7a-7522-4735-9d27-2ac1237c8f03-cnibin\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.911215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.910036    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9608f8a5-7473-4876-a9de-fd6b06753e13-ovnkube-config\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.912512 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.912496    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9608f8a5-7473-4876-a9de-fd6b06753e13-ovn-node-metrics-cert\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.913483 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.913469    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rw8q\" (UniqueName: \"kubernetes.io/projected/95595c7a-7522-4735-9d27-2ac1237c8f03-kube-api-access-4rw8q\") pod \"multus-q7xh8\" (UID: \"95595c7a-7522-4735-9d27-2ac1237c8f03\") " pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:08.914530 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.914512    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2mv\" (UniqueName: \"kubernetes.io/projected/8e9a83a3-4849-4baf-a9a7-781bf7974983-kube-api-access-5k2mv\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:08.915404 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.915384    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwv8\" (UniqueName: \"kubernetes.io/projected/9608f8a5-7473-4876-a9de-fd6b06753e13-kube-api-access-6bwv8\") pod \"ovnkube-node-4m68j\" (UID: \"9608f8a5-7473-4876-a9de-fd6b06753e13\") " pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:08.915482 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.915443    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756kb\" (UniqueName: \"kubernetes.io/projected/0c6c57df-4530-4a13-9437-4256b6ccff0d-kube-api-access-756kb\") pod \"multus-additional-cni-plugins-chk96\" (UID: \"0c6c57df-4530-4a13-9437-4256b6ccff0d\") " pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:08.917705 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.917691    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:08.917786 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.917709    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:08.917786 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.917721    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:08.917786 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:08.917781    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:09.417766615 +0000 UTC m=+3.101250510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:08.919398 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.919383    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpcs\" (UniqueName: \"kubernetes.io/projected/bcfc626c-0f15-49dd-85e8-4a65410fee34-kube-api-access-qwpcs\") pod \"aws-ebs-csi-driver-node-dv7gf\" (UID: \"bcfc626c-0f15-49dd-85e8-4a65410fee34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:08.919586 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.919570    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z597\" (UniqueName: \"kubernetes.io/projected/65b00689-c10f-4b94-a41e-292d4ed16225-kube-api-access-5z597\") pod \"iptables-alerter-wqq4l\" (UID: \"65b00689-c10f-4b94-a41e-292d4ed16225\") " pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:08.994883 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.994834    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:08.999646 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:08.999633    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qbsrl"
Jul 01 20:46:09.003743 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.003725    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26c1158_7a44_4ec3_afc7_f3b0c2c4a047.slice/crio-a28cada84d19a9d5d5008026ba81f7d8b8da73c37cb2dd31423e9b3a639a3246 WatchSource:0}: Error finding container a28cada84d19a9d5d5008026ba81f7d8b8da73c37cb2dd31423e9b3a639a3246: Status 404 returned error can't find the container with id a28cada84d19a9d5d5008026ba81f7d8b8da73c37cb2dd31423e9b3a639a3246
Jul 01 20:46:09.006786 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.006758    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qn9x"
Jul 01 20:46:09.007650 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.007632    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd06c46_3ee3_4b0b_b239_9b2356dd099e.slice/crio-fa5b604024d262ba5934dce14267ab80ddb3f69b5642a8194a1769ef7ca38042 WatchSource:0}: Error finding container fa5b604024d262ba5934dce14267ab80ddb3f69b5642a8194a1769ef7ca38042: Status 404 returned error can't find the container with id fa5b604024d262ba5934dce14267ab80ddb3f69b5642a8194a1769ef7ca38042
Jul 01 20:46:09.009863 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.009850    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24krb"
Jul 01 20:46:09.015524 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.015509    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78922f2f_7ca3_468e_ae75_5a9cb7ca1563.slice/crio-cd43beca4a4138201cb476b0267e39e6cb52cc85d7a4b87ef27686ec079d6c2f WatchSource:0}: Error finding container cd43beca4a4138201cb476b0267e39e6cb52cc85d7a4b87ef27686ec079d6c2f: Status 404 returned error can't find the container with id cd43beca4a4138201cb476b0267e39e6cb52cc85d7a4b87ef27686ec079d6c2f
Jul 01 20:46:09.017889 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.016899    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-chk96"
Jul 01 20:46:09.019173 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.019155    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153889c1_36ed_424f_ad2e_4e3d98b84921.slice/crio-bbde626cbab1b311352fe165f95f7ab4aaa5fa3b88a23b6ea9da8913d5f70764 WatchSource:0}: Error finding container bbde626cbab1b311352fe165f95f7ab4aaa5fa3b88a23b6ea9da8913d5f70764: Status 404 returned error can't find the container with id bbde626cbab1b311352fe165f95f7ab4aaa5fa3b88a23b6ea9da8913d5f70764
Jul 01 20:46:09.021681 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.021665    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q7xh8"
Jul 01 20:46:09.026600 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.026538    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf"
Jul 01 20:46:09.026744 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.026723    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6c57df_4530_4a13_9437_4256b6ccff0d.slice/crio-f548fb2acc5e8cec997faa90ee8687cb6dd8c54542154d1741d7fcb690935673 WatchSource:0}: Error finding container f548fb2acc5e8cec997faa90ee8687cb6dd8c54542154d1741d7fcb690935673: Status 404 returned error can't find the container with id f548fb2acc5e8cec997faa90ee8687cb6dd8c54542154d1741d7fcb690935673
Jul 01 20:46:09.030671 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.030642    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95595c7a_7522_4735_9d27_2ac1237c8f03.slice/crio-0d252c9ff67a98811857cb1be0f405014e647e7743f26144090ed20cbffeb6b1 WatchSource:0}: Error finding container 0d252c9ff67a98811857cb1be0f405014e647e7743f26144090ed20cbffeb6b1: Status 404 returned error can't find the container with id 0d252c9ff67a98811857cb1be0f405014e647e7743f26144090ed20cbffeb6b1
Jul 01 20:46:09.032583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.032565    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wqq4l"
Jul 01 20:46:09.035033 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.035013    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc626c_0f15_49dd_85e8_4a65410fee34.slice/crio-da3d1d6c13696bdb2e82f2ccdb7d11fa66f92b08874d3359b68521dc9b7d7d50 WatchSource:0}: Error finding container da3d1d6c13696bdb2e82f2ccdb7d11fa66f92b08874d3359b68521dc9b7d7d50: Status 404 returned error can't find the container with id da3d1d6c13696bdb2e82f2ccdb7d11fa66f92b08874d3359b68521dc9b7d7d50
Jul 01 20:46:09.036459 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.036307    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:09.042498 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.042473    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b00689_c10f_4b94_a41e_292d4ed16225.slice/crio-745908ee3f209f20f294a6b0b7075937c9fb6efb9a0401f93b19d194e2495eb9 WatchSource:0}: Error finding container 745908ee3f209f20f294a6b0b7075937c9fb6efb9a0401f93b19d194e2495eb9: Status 404 returned error can't find the container with id 745908ee3f209f20f294a6b0b7075937c9fb6efb9a0401f93b19d194e2495eb9
Jul 01 20:46:09.045793 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:09.045775    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9608f8a5_7473_4876_a9de_fd6b06753e13.slice/crio-a88607cee2d934bcdb744eb2e3bd12985bdae772e393ade4b6e4041e8b08703c WatchSource:0}: Error finding container a88607cee2d934bcdb744eb2e3bd12985bdae772e393ade4b6e4041e8b08703c: Status 404 returned error can't find the container with id a88607cee2d934bcdb744eb2e3bd12985bdae772e393ade4b6e4041e8b08703c
Jul 01 20:46:09.412115 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.412078    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:09.412283 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.412201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:09.412283 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.412263    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:09.412565 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.412322    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:09.412565 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.412339    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.412318553 +0000 UTC m=+4.095802444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:09.412565 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.412406    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.412387681 +0000 UTC m=+4.095871566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:09.513855 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.513508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:09.513855 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.513733    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:09.513855 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.513754    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:09.513855 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.513766    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:09.513855 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:09.513822    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.513805641 +0000 UTC m=+4.197289537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:09.735454 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.735361    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:08 +0000 UTC" deadline="2028-03-23 18:12:51.436096714 +0000 UTC"
Jul 01 20:46:09.735454 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.735400    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15141h26m41.70070109s"
Jul 01 20:46:09.776561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.776524    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"a88607cee2d934bcdb744eb2e3bd12985bdae772e393ade4b6e4041e8b08703c"}
Jul 01 20:46:09.778118 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.778039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wqq4l" event={"ID":"65b00689-c10f-4b94-a41e-292d4ed16225","Type":"ContainerStarted","Data":"745908ee3f209f20f294a6b0b7075937c9fb6efb9a0401f93b19d194e2495eb9"}
Jul 01 20:46:09.780185 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.780107    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf" event={"ID":"bcfc626c-0f15-49dd-85e8-4a65410fee34","Type":"ContainerStarted","Data":"da3d1d6c13696bdb2e82f2ccdb7d11fa66f92b08874d3359b68521dc9b7d7d50"}
Jul 01 20:46:09.783429 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.783406    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q7xh8" event={"ID":"95595c7a-7522-4735-9d27-2ac1237c8f03","Type":"ContainerStarted","Data":"0d252c9ff67a98811857cb1be0f405014e647e7743f26144090ed20cbffeb6b1"}
Jul 01 20:46:09.785236 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.785211    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24krb" event={"ID":"153889c1-36ed-424f-ad2e-4e3d98b84921","Type":"ContainerStarted","Data":"bbde626cbab1b311352fe165f95f7ab4aaa5fa3b88a23b6ea9da8913d5f70764"}
Jul 01 20:46:09.786807 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.786788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kdd8w" event={"ID":"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047","Type":"ContainerStarted","Data":"a28cada84d19a9d5d5008026ba81f7d8b8da73c37cb2dd31423e9b3a639a3246"}
Jul 01 20:46:09.790545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.790480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerStarted","Data":"f548fb2acc5e8cec997faa90ee8687cb6dd8c54542154d1741d7fcb690935673"}
Jul 01 20:46:09.792590 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.792527    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qn9x" event={"ID":"78922f2f-7ca3-468e-ae75-5a9cb7ca1563","Type":"ContainerStarted","Data":"cd43beca4a4138201cb476b0267e39e6cb52cc85d7a4b87ef27686ec079d6c2f"}
Jul 01 20:46:09.793964 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:09.793921    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qbsrl" event={"ID":"add06c46-3ee3-4b0b-b239-9b2356dd099e","Type":"ContainerStarted","Data":"fa5b604024d262ba5934dce14267ab80ddb3f69b5642a8194a1769ef7ca38042"}
Jul 01 20:46:10.421068 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.421031    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:10.421243 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.421116    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:10.421353 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.421333    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:10.421418 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.421411    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:12.421386636 +0000 UTC m=+6.104870521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:10.421483 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.421477    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:10.421531 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.421512    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:12.421498644 +0000 UTC m=+6.104982526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:10.521515 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.521481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:10.521729 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.521712    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:10.521825 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.521738    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:10.521825 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.521750    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:10.521825 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.521808    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:12.521791055 +0000 UTC m=+6.205274948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:10.736048 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.735963    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:08 +0000 UTC" deadline="2028-01-13 01:29:06.211953546 +0000 UTC"
Jul 01 20:46:10.736048 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.736002    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13444h42m55.47595655s"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.769724    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.769860    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.769949    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.770033    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:10.770091    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:10.770202 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:10.770155    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:12.438318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.438266    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:12.438792 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.438359    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:12.438792 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.438481    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:12.438792 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.438543    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:16.438524393 +0000 UTC m=+10.122008278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:12.438792 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.438620    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:12.438792 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.438655    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:16.438644034 +0000 UTC m=+10.122127919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:12.539201 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.539164    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:12.539366 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.539349    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:12.539449 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.539377    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:12.539449 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.539390    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:12.539548 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.539451    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:16.539431907 +0000 UTC m=+10.222915794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:12.768509 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.768385    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:12.768509 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.768419    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:12.768736 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.768522    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:12.769209 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.768904    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:12.769209 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:12.768964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:12.769209 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:12.769091    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:14.767498    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:14.767634    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:14.767637    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:14.767722    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:14.767750    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:14.768014 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:14.767815    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:16.474635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.474539    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:16.474635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.474620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:16.475261 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.474704    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:16.475261 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.474723    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:16.475261 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.474777    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:24.474756772 +0000 UTC m=+18.158240668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:16.475261 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.474797    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:24.474786563 +0000 UTC m=+18.158270451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:16.575202 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.575168    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:16.575375 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.575349    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:16.575375 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.575374    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:16.575501 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.575387    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:16.575501 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.575453    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:24.575435027 +0000 UTC m=+18.258918931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:16.768766 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.768697    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:16.770423 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.769058    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:16.770423 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.769519    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:16.770423 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.769624    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:16.770423 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:16.769809    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:16.770423 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:16.769899    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:18.767409 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:18.767374    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:18.767893 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:18.767507    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:18.767893 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:18.767863    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:18.768093 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:18.767966    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:18.768093 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:18.767993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:18.768093 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:18.768085    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:20.767632 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:20.767410    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:20.768101 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:20.767418    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:20.768101 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:20.767731    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:20.768101 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:20.767487    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:20.768101 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:20.767812    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:20.768101 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:20.767911    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:22.767906 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:22.767842    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:22.768396 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:22.767852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:22.768396 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:22.767978    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:22.768396 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:22.768097    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:22.768396 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:22.768129    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:22.768396 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:22.768208    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:24.537999 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.537960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:24.538448 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.538023    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:24.538448 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.538119    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:24.538448 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.538168    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:40.538154497 +0000 UTC m=+34.221638378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:24.538448 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.538210    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:24.538448 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.538285    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:40.538258051 +0000 UTC m=+34.221741932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:24.638486 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.638456    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:24.638655 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.638575    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:24.638655 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.638596    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:24.638655 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.638608    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:24.638816 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.638659    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:40.638646041 +0000 UTC m=+34.322129922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:24.767585 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.767552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:24.767764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.767556    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:24.767764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:24.767693    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:24.767764 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.767688    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:24.767913 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.767768    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:24.767913 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:24.767830    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:26.768048 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:26.768022    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:26.768490 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:26.768135    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:26.768490 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:26.768212    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:26.768490 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:26.768308    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:26.768490 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:26.768341    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:26.768490 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:26.768424    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:28.767428 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.767235    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:28.768138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.767258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:28.768138 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:28.767490    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:28.768138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.767297    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:28.768138 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:28.767536    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:28.768138 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:28.767612    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:28.848466 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.848435    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:46:28.848883 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.848849    2554 generic.go:356] "Generic (PLEG): container finished" podID="9608f8a5-7473-4876-a9de-fd6b06753e13" containerID="ea963c556300037021dc606d776e909ca16053d95e60fcf5e3d31d8f77d9651b" exitCode=1
Jul 01 20:46:28.849027 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.848930    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"44143eb1d5f1b4a6efdfb9acbaf8baeada32bf86e81f79ade1fef013d95d486d"}
Jul 01 20:46:28.849083 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.849053    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"047368f2349734445d999947ec5d9a0718aeca5fd3ffc657b9395e94f8cbb466"}
Jul 01 20:46:28.849136 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.849104    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"caaf65ad10ed9a8b960812ae2f908a4c605dd4fb4a6fb68498aea02b3b7ad005"}
Jul 01 20:46:28.849192 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.849150    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"041b9439622cd9e249c9a4b80b74126fcf71e8a6405c21222fb9caef95f70981"}
Jul 01 20:46:28.849244 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.849196    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerDied","Data":"ea963c556300037021dc606d776e909ca16053d95e60fcf5e3d31d8f77d9651b"}
Jul 01 20:46:28.849293 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.849244    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"10d90520536bad69cf12f92346baee2ad95309de287cf1be6a49d5caca6a9e28"}
Jul 01 20:46:28.850620 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.850595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf" event={"ID":"bcfc626c-0f15-49dd-85e8-4a65410fee34","Type":"ContainerStarted","Data":"5c943c3a06b7db28ceff979f884906c50c095ee5064ecaf94dee84400574a5d6"}
Jul 01 20:46:28.852440 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.852417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q7xh8" event={"ID":"95595c7a-7522-4735-9d27-2ac1237c8f03","Type":"ContainerStarted","Data":"6f5fd5552c1e2e70259a49d05c7dac8ffdea7a05ebce9d82339ee121881bfd7b"}
Jul 01 20:46:28.854006 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.853978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24krb" event={"ID":"153889c1-36ed-424f-ad2e-4e3d98b84921","Type":"ContainerStarted","Data":"898dc8b0c2de00a2e761ac5ded47b2feafda9e9a0d3b5abedfa06c3d1996c3b5"}
Jul 01 20:46:28.855836 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.855807    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kdd8w" event={"ID":"b26c1158-7a44-4ec3-afc7-f3b0c2c4a047","Type":"ContainerStarted","Data":"9faa035a47f4dae198c6d74172cd138e1a29a34909fc09ca883bedba002ea8ec"}
Jul 01 20:46:28.857487 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.857464    2554 generic.go:356] "Generic (PLEG): container finished" podID="7c21388212b058e3b39cb3124c51bb32" containerID="08b096ec9469daeb27c0ad3d4ada03e62e01181cb851775399c7e3e7f7efd0de" exitCode=0
Jul 01 20:46:28.857598 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.857542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"7c21388212b058e3b39cb3124c51bb32","Type":"ContainerDied","Data":"08b096ec9469daeb27c0ad3d4ada03e62e01181cb851775399c7e3e7f7efd0de"}
Jul 01 20:46:28.861551 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.861527    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="ddfb077523deb5096299d08c10e9fa8df62cc6a418b8ccf28b5daa18924cd3ef" exitCode=0
Jul 01 20:46:28.861648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.861602    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"ddfb077523deb5096299d08c10e9fa8df62cc6a418b8ccf28b5daa18924cd3ef"}
Jul 01 20:46:28.863591 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.863566    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qn9x" event={"ID":"78922f2f-7ca3-468e-ae75-5a9cb7ca1563","Type":"ContainerStarted","Data":"7df21343ab4976bdd5b24ea0a5318b871f891f4f61ba65718286c8145893aab3"}
Jul 01 20:46:28.865183 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.865156    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qbsrl" event={"ID":"add06c46-3ee3-4b0b-b239-9b2356dd099e","Type":"ContainerStarted","Data":"1fab44af12ff6653b3e0c43c2201116554e48ed491e015a956cdb9064063376e"}
Jul 01 20:46:28.866601 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.866579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"0d191265e6bbab3f8d2d9679a92fcb37","Type":"ContainerStarted","Data":"ff5fffd6fe363ef1f9762c7f4f5c856b83881374cbaf7107b72ff48909ba6968"}
Jul 01 20:46:28.876747 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.876705    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q7xh8" podStartSLOduration=3.026064472 podStartE2EDuration="21.876693707s" totalImagesPullingTime="18.850629235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.876042547 +0000 UTC m=+22.559526452" watchObservedRunningTime="2026-07-01 20:46:28.876693707 +0000 UTC m=+22.560177610"
Jul 01 20:46:28.894583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.894544    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kdd8w" podStartSLOduration=3.172808022 podStartE2EDuration="21.894530303s" totalImagesPullingTime="18.721722281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.894299979 +0000 UTC m=+22.577783881" watchObservedRunningTime="2026-07-01 20:46:28.894530303 +0000 UTC m=+22.578014207"
Jul 01 20:46:28.936340 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.936084    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qbsrl" podStartSLOduration=3.232066022 podStartE2EDuration="21.936065475s" totalImagesPullingTime="18.703999453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.915679766 +0000 UTC m=+22.599163671" watchObservedRunningTime="2026-07-01 20:46:28.936065475 +0000 UTC m=+22.619549380"
Jul 01 20:46:28.936481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.936440    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7qn9x" podStartSLOduration=3.229177004 podStartE2EDuration="21.936431921s" totalImagesPullingTime="18.707254917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.93572563 +0000 UTC m=+22.619209535" watchObservedRunningTime="2026-07-01 20:46:28.936431921 +0000 UTC m=+22.619915827"
Jul 01 20:46:28.959542 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:28.959499    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" podStartSLOduration=21.959486711 podStartE2EDuration="21.959486711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.959359713 +0000 UTC m=+22.642843617" watchObservedRunningTime="2026-07-01 20:46:28.959486711 +0000 UTC m=+22.642970614"
Jul 01 20:46:29.041192 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.041106    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-24krb" podStartSLOduration=3.586681742 podStartE2EDuration="22.041092174s" totalImagesPullingTime="18.454410432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:28.98810426 +0000 UTC m=+22.671588165" watchObservedRunningTime="2026-07-01 20:46:29.041092174 +0000 UTC m=+22.724576091"
Jul 01 20:46:29.643796 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.643772    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 20:46:29.774146 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.774021    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T20:46:29.643792812Z","UUID":"6faf5cae-a68f-489e-8ded-3337950414be","Handler":null,"Name":"","Endpoint":""}
Jul 01 20:46:29.775366 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.775348    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 20:46:29.775473 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.775379    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 20:46:29.870774 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.870650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wqq4l" event={"ID":"65b00689-c10f-4b94-a41e-292d4ed16225","Type":"ContainerStarted","Data":"d7d2a1d46e171de6e5c798722d60d1d435d6b2cee2d4dae7503a047d2d9b58fa"}
Jul 01 20:46:29.872805 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.872775    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf" event={"ID":"bcfc626c-0f15-49dd-85e8-4a65410fee34","Type":"ContainerStarted","Data":"2b39f618494910db0d980d91b72422d8ca8159f38f91e18c5d484d5603f3ff0a"}
Jul 01 20:46:29.874741 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.874703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"7c21388212b058e3b39cb3124c51bb32","Type":"ContainerStarted","Data":"663a17e6ef53842d43f83c4384692ffdceab2d4419331f76562a7eb4a1779005"}
Jul 01 20:46:29.912559 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.912520    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" podStartSLOduration=22.912505399 podStartE2EDuration="22.912505399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:29.91203526 +0000 UTC m=+23.595519164" watchObservedRunningTime="2026-07-01 20:46:29.912505399 +0000 UTC m=+23.595989510"
Jul 01 20:46:29.912716 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.912638    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wqq4l" podStartSLOduration=8.65227049 podStartE2EDuration="22.912630806s" totalImagesPullingTime="14.260360316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:29.888520866 +0000 UTC m=+23.572004771" watchObservedRunningTime="2026-07-01 20:46:29.912630806 +0000 UTC m=+23.596114712"
Jul 01 20:46:29.956350 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.956322    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:29.957058 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:29.957039    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:30.767954 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.767915    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:30.767954 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.767925    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:30.768150 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:30.768042    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:30.768150 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.768087    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:30.768257 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:30.768176    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:30.768313 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:30.768262    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:30.881050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.880986    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:46:30.881497 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.881460    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"5a4fa12913580d7ae6cb62148bd838d09f2cf18f7d7c91b36df453b0e2d63235"}
Jul 01 20:46:30.881924 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.881891    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:30.882357 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:30.882337    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kdd8w"
Jul 01 20:46:31.885523 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:31.885469    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf" event={"ID":"bcfc626c-0f15-49dd-85e8-4a65410fee34","Type":"ContainerStarted","Data":"c791628bc4a55bd4b8eb26f58f171087cb755badb771aace477fdaf8b169cdba"}
Jul 01 20:46:31.909617 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:31.909576    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7gf" podStartSLOduration=3.518988672 podStartE2EDuration="24.909564044s" totalImagesPullingTime="21.390575372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:31.909464365 +0000 UTC m=+25.592948269" watchObservedRunningTime="2026-07-01 20:46:31.909564044 +0000 UTC m=+25.593047947"
Jul 01 20:46:32.767099 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:32.767064    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:32.767280 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:32.767073    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:32.767280 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:32.767200    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:32.767390 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:32.767074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:32.767390 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:32.767288    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:32.767485 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:32.767422    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:33.892531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:33.892366    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:46:33.893225 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:33.892886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"646e965ae630f1cd90d996632d8df5e604192933f20f397870f86e2e234e277f"}
Jul 01 20:46:33.893225 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:33.893154    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:33.893377 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:33.893361    2554 scope.go:117] "RemoveContainer" containerID="ea963c556300037021dc606d776e909ca16053d95e60fcf5e3d31d8f77d9651b"
Jul 01 20:46:33.909895 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:33.909876    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:34.767482 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.767210    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:34.767682 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:34.767518    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:34.767682 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.767260    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:34.767682 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:34.767638    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:34.767682 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.767210    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:34.767884 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:34.767706    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:34.897108 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.897068    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="5ab071b9675c8d4ae3fad7bbdfc80afc5202aaf60c0f2bb97b4784cab5cccc04" exitCode=0
Jul 01 20:46:34.897499 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.897130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"5ab071b9675c8d4ae3fad7bbdfc80afc5202aaf60c0f2bb97b4784cab5cccc04"}
Jul 01 20:46:34.900811 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.900796    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:46:34.901132 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.901113    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" event={"ID":"9608f8a5-7473-4876-a9de-fd6b06753e13","Type":"ContainerStarted","Data":"be7dbfe650649857fba389174c51f025ac9b62aa7b74176a5b6f569783072c7d"}
Jul 01 20:46:34.901381 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.901363    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:34.901484 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.901422    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:34.915489 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.915473    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:46:34.976639 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:34.976597    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j" podStartSLOduration=9.254380684000001 podStartE2EDuration="27.976583925s" totalImagesPullingTime="18.722203241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:34.976068339 +0000 UTC m=+28.659552243" watchObservedRunningTime="2026-07-01 20:46:34.976583925 +0000 UTC m=+28.660067829"
Jul 01 20:46:35.632531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.632499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9scdv"]
Jul 01 20:46:35.632770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.632610    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:35.632770 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:35.632715    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:35.634511 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.634486    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbm9x"]
Jul 01 20:46:35.634635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.634580    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:35.634688 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:35.634676    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:35.637494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.637474    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7fwz5"]
Jul 01 20:46:35.637593 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:35.637546    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:35.637645 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:35.637623    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:36.768434 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:36.768407    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:36.768895 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:36.768520    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:36.768895 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:36.768609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:36.768895 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:36.768724    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:36.768895 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:36.768734    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:36.768895 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:36.768800    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:36.906630 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:36.906601    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="183de634a1fbe0d9433b366be46ef7baf9da02d3b332293f1cd403884a5f4362" exitCode=0
Jul 01 20:46:36.906780 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:36.906683    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"183de634a1fbe0d9433b366be46ef7baf9da02d3b332293f1cd403884a5f4362"}
Jul 01 20:46:38.767138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:38.766920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:38.767559 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:38.766920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:38.767559 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:38.767214    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbm9x" podUID="05f7d919-ddfa-438a-9dd8-e5c6d4ed8498"
Jul 01 20:46:38.767559 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:38.766920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:38.767559 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:38.767313    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9scdv" podUID="e4f34b82-ccc9-474f-8fd5-f332edbdacc6"
Jul 01 20:46:38.767559 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:38.767362    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7fwz5" podUID="8e9a83a3-4849-4baf-a9a7-781bf7974983"
Jul 01 20:46:38.913623 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:38.913539    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="02b4cdae999ceee9251431646cafe5a6729e85106b4bb90c07ac280077437b06" exitCode=0
Jul 01 20:46:38.913757 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:38.913614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"02b4cdae999ceee9251431646cafe5a6729e85106b4bb90c07ac280077437b06"}
Jul 01 20:46:40.552978 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.552926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:40.553462 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.553101    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:40.553462 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.553106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:40.553462 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.553182    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:40.553462 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.553185    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.553154663 +0000 UTC m=+66.236638558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:40.553462 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.553246    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret podName:05f7d919-ddfa-438a-9dd8-e5c6d4ed8498 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.553232841 +0000 UTC m=+66.236716725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret") pod "global-pull-secret-syncer-cbm9x" (UID: "05f7d919-ddfa-438a-9dd8-e5c6d4ed8498") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:40.624117 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.624085    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeReady"
Jul 01 20:46:40.624279 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.624237    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 20:46:40.654568 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.654543    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:40.654698 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.654662    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:40.654698 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.654677    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:40.654698 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.654686    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mqt9c for pod openshift-network-diagnostics/network-check-target-9scdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:40.654820 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.654740    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c podName:e4f34b82-ccc9-474f-8fd5-f332edbdacc6 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.654727636 +0000 UTC m=+66.338211518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mqt9c" (UniqueName: "kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c") pod "network-check-target-9scdv" (UID: "e4f34b82-ccc9-474f-8fd5-f332edbdacc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:40.681557 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.681529    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:46:40.684163 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.684146    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.687425 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.687380    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 20:46:40.687527 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.687438    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 20:46:40.687527 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.687443    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgzdr\""
Jul 01 20:46:40.687958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.687926    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 20:46:40.698955 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.698917    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 20:46:40.699969 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.699928    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jzns2"]
Jul 01 20:46:40.702810 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.702795    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.704738 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.704721    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:46:40.708688 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.708649    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 20:46:40.708977 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.708960    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4r2bh\""
Jul 01 20:46:40.709102 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.709087    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7dq69"]
Jul 01 20:46:40.710827 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.710808    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 20:46:40.711959 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.711925    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.715764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.715743    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 20:46:40.715764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.715756    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w7rxr\""
Jul 01 20:46:40.716366 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.716348    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 20:46:40.716450 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.716389    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 20:46:40.725057 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.725038    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzns2"]
Jul 01 20:46:40.730680 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.730666    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dq69"]
Jul 01 20:46:40.767435 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.767407    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:46:40.767572 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.767407    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:46:40.767634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.767417    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:46:40.770558 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.770531    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tg27v\""
Jul 01 20:46:40.770844 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.770825    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xhskk\""
Jul 01 20:46:40.771636 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.771544    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:46:40.771636 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.771586    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:46:40.771811 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.771666    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:46:40.771811 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.771806    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:46:40.856023 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.855993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856191 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856191 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f752519e-26b4-4130-b085-0b3b60b7a89e-config-volume\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.856191 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.856312 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856207    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856312 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856235    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856312 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qlsv\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856420 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856338    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f752519e-26b4-4130-b085-0b3b60b7a89e-tmp-dir\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.856420 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856420 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856420 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632a983-a19e-47ef-a1d5-ff80bf434faf-cert\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.856535 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqzl\" (UniqueName: \"kubernetes.io/projected/f752519e-26b4-4130-b085-0b3b60b7a89e-kube-api-access-xvqzl\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.856535 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.856535 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.856460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvktx\" (UniqueName: \"kubernetes.io/projected/6632a983-a19e-47ef-a1d5-ff80bf434faf-kube-api-access-mvktx\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.957648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.957648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957603    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvktx\" (UniqueName: \"kubernetes.io/projected/6632a983-a19e-47ef-a1d5-ff80bf434faf-kube-api-access-mvktx\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.957648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957647    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.957903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.957903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f752519e-26b4-4130-b085-0b3b60b7a89e-config-volume\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.957903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957741    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.957903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.957768    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.957903 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.957859    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:40.958174 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.957922    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:46:41.457902407 +0000 UTC m=+35.141386292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:40.958174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.958174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958060    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qlsv\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.958174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958135    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.958174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958145    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f752519e-26b4-4130-b085-0b3b60b7a89e-tmp-dir\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.958179    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.958192    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:40.958232    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:46:41.458219189 +0000 UTC m=+35.141703089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958270    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632a983-a19e-47ef-a1d5-ff80bf434faf-cert\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.958424 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqzl\" (UniqueName: \"kubernetes.io/projected/f752519e-26b4-4130-b085-0b3b60b7a89e-kube-api-access-xvqzl\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.958949 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958901    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f752519e-26b4-4130-b085-0b3b60b7a89e-tmp-dir\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.959049 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.958960    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.959473 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.959449    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f752519e-26b4-4130-b085-0b3b60b7a89e-config-volume\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.959690 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.959667    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.962574 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.962461    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.962574 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.962536    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.962712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.962584    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632a983-a19e-47ef-a1d5-ff80bf434faf-cert\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.968048 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.968002    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:40.968396 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.968369    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqzl\" (UniqueName: \"kubernetes.io/projected/f752519e-26b4-4130-b085-0b3b60b7a89e-kube-api-access-xvqzl\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:40.968499 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.968465    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvktx\" (UniqueName: \"kubernetes.io/projected/6632a983-a19e-47ef-a1d5-ff80bf434faf-kube-api-access-mvktx\") pod \"ingress-canary-7dq69\" (UID: \"6632a983-a19e-47ef-a1d5-ff80bf434faf\") " pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:40.968802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:40.968759    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qlsv\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:41.020409 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.020386    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dq69"
Jul 01 20:46:41.141767 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.141732    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:41.167638 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.167610    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dq69"]
Jul 01 20:46:41.173012 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:46:41.172984    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6632a983_a19e_47ef_a1d5_ff80bf434faf.slice/crio-f188b1cb169a36194e7fb5ee652182fc54968944ef51037f0a0f7b509c499bde WatchSource:0}: Error finding container f188b1cb169a36194e7fb5ee652182fc54968944ef51037f0a0f7b509c499bde: Status 404 returned error can't find the container with id f188b1cb169a36194e7fb5ee652182fc54968944ef51037f0a0f7b509c499bde
Jul 01 20:46:41.243268 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.243203    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:41.462601 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.462564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:41.462601 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.462601    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:41.462810 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:41.462743    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:41.462810 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:41.462764    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:41.462810 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:41.462777    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:41.462964 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:41.462814    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:46:42.462796937 +0000 UTC m=+36.146280821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:41.462964 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:41.462834    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:46:42.462825916 +0000 UTC m=+36.146309805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:41.557149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.557075    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:41.654394 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.654181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:41.921746 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:41.921695    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dq69" event={"ID":"6632a983-a19e-47ef-a1d5-ff80bf434faf","Type":"ContainerStarted","Data":"f188b1cb169a36194e7fb5ee652182fc54968944ef51037f0a0f7b509c499bde"}
Jul 01 20:46:42.470467 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:42.470429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:42.470480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:42.470555    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:42.470601    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:42.470618    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:42.470636    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.470619778 +0000 UTC m=+38.154103664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:42.470678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:42.470670    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.470651575 +0000 UTC m=+38.154135480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:43.929628 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:43.929599    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="03198fcfec183a5bb34fda1fb3c3aef3fc3403e25b3301f9780c03a1f447beee" exitCode=0
Jul 01 20:46:43.930017 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:43.929639    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"03198fcfec183a5bb34fda1fb3c3aef3fc3403e25b3301f9780c03a1f447beee"}
Jul 01 20:46:44.486751 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:44.486722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:44.486901 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:44.486759    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:44.486901 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:44.486865    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:44.486901 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:44.486882    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:44.486901 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:44.486892    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:44.487056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:44.486924    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:46:48.486910506 +0000 UTC m=+42.170394387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:44.487056 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:44.486953    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:46:48.486946464 +0000 UTC m=+42.170430346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:44.935259 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:44.935224    2554 generic.go:356] "Generic (PLEG): container finished" podID="0c6c57df-4530-4a13-9437-4256b6ccff0d" containerID="0212548b923c51ba3c23479640770edde0299f027255ea768996f441d25d0c98" exitCode=0
Jul 01 20:46:44.935626 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:44.935265    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerDied","Data":"0212548b923c51ba3c23479640770edde0299f027255ea768996f441d25d0c98"}
Jul 01 20:46:45.070170 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:45.070136    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:45.940995 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:45.940961    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-chk96" event={"ID":"0c6c57df-4530-4a13-9437-4256b6ccff0d","Type":"ContainerStarted","Data":"286f1b658c900a78bcbe78d3b2767c9d6f84ee6a03a13f5f801a3b28adade095"}
Jul 01 20:46:45.973767 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:45.973718    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-chk96" podStartSLOduration=13.273931098 podStartE2EDuration="38.973703574s" totalImagesPullingTime="25.699772476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:45.972170164 +0000 UTC m=+39.655654059" watchObservedRunningTime="2026-07-01 20:46:45.973703574 +0000 UTC m=+39.657187479"
Jul 01 20:46:46.101685 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:46.101651    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.202539 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:46.202513    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.945150 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:46.945109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dq69" event={"ID":"6632a983-a19e-47ef-a1d5-ff80bf434faf","Type":"ContainerStarted","Data":"154ba79907255eab61a8d4d903a58bef1fb8a44ff2e5472971dafda8d780d10e"}
Jul 01 20:46:46.964291 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:46.964244    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7dq69" podStartSLOduration=1.97386706 podStartE2EDuration="6.964229661s" totalImagesPullingTime="4.990362601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:46.963414171 +0000 UTC m=+40.646898074" watchObservedRunningTime="2026-07-01 20:46:46.964229661 +0000 UTC m=+40.647713601"
Jul 01 20:46:48.519691 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:48.519656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:48.519693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:48.519765    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:48.519822    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:48.519833    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:48.519849    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:46:56.519830117 +0000 UTC m=+50.203314012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:48.520071 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:48.519868    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:46:56.51985754 +0000 UTC m=+50.203341422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:56.573453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:56.573412    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:46:56.573453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:46:56.573450    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:46:56.573926 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:56.573581    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:56.573926 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:56.573593    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:46:56.573926 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:56.573591    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:56.573926 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:56.573641    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.573627497 +0000 UTC m=+66.257111380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:46:56.573926 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:46:56.573665    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.573645741 +0000 UTC m=+66.257129641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:47:06.920535 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:06.920508    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4m68j"
Jul 01 20:47:11.341611 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.341580    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"]
Jul 01 20:47:11.347104 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.347081    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.350149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.350131    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 20:47:11.350637 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.350615    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 20:47:11.351133 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.351118    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4tmdw\""
Jul 01 20:47:11.366522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.366501    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"]
Jul 01 20:47:11.368146 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.368121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2008ae7-08d6-4147-944b-492d48480330-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.368234 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.368147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.415011 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.414987    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"]
Jul 01 20:47:11.417118 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.417100    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.421378 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.421356    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:47:11.421487 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.421467    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 01 20:47:11.421531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.421479    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2xvms\""
Jul 01 20:47:11.421572 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.421532    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 01 20:47:11.423464 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.423448    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 01 20:47:11.439447 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.439425    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"]
Jul 01 20:47:11.469063 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2008ae7-08d6-4147-944b-492d48480330-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.469182 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469064    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.469182 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469089    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/022a57d4-a679-49c2-98ac-2c6a12178ff8-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.469182 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts25\" (UniqueName: \"kubernetes.io/projected/022a57d4-a679-49c2-98ac-2c6a12178ff8-kube-api-access-2ts25\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.469182 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/022a57d4-a679-49c2-98ac-2c6a12178ff8-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.469355 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:11.469181    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:11.469355 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:11.469244    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:11.969229329 +0000 UTC m=+65.652713210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:11.469678 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.469661    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2008ae7-08d6-4147-944b-492d48480330-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.518050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.518027    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"]
Jul 01 20:47:11.520439 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.520421    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"]
Jul 01 20:47:11.520544 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.520534    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.522702 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.522682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.523683 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.523663    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-4bj29"]
Jul 01 20:47:11.524813 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.524785    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 01 20:47:11.524924 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.524909    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 01 20:47:11.524990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.524968    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:47:11.525720 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.525707    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-skh4d\""
Jul 01 20:47:11.526011 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.525997    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29"
Jul 01 20:47:11.527543 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.527525    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 01 20:47:11.530878 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.530862    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 01 20:47:11.539007 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.538989    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 01 20:47:11.539135 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.539117    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 01 20:47:11.539262 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.539245    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 01 20:47:11.539339 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.539262    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rqm5z\""
Jul 01 20:47:11.539545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.539460    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-g9q5v\""
Jul 01 20:47:11.544257 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.544241    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:47:11.553118 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.553100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"]
Jul 01 20:47:11.553908 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.553889    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-4bj29"]
Jul 01 20:47:11.568775 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.568753    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"]
Jul 01 20:47:11.569597 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569573    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts25\" (UniqueName: \"kubernetes.io/projected/022a57d4-a679-49c2-98ac-2c6a12178ff8-kube-api-access-2ts25\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.569691 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569624    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-trusted-ca\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.569691 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-serving-cert\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.569691 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569672    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.569839 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569695    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r729\" (UniqueName: \"kubernetes.io/projected/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-kube-api-access-8r729\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.569839 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-config\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.569959 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569879    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/022a57d4-a679-49c2-98ac-2c6a12178ff8-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.569959 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcljc\" (UniqueName: \"kubernetes.io/projected/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-kube-api-access-wcljc\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.570138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569971    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qrd\" (UniqueName: \"kubernetes.io/projected/182b1f4c-e9ad-4de8-86a9-fddcadd3952f-kube-api-access-52qrd\") pod \"network-check-source-56499946bc-4bj29\" (UID: \"182b1f4c-e9ad-4de8-86a9-fddcadd3952f\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29"
Jul 01 20:47:11.570138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.569999    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/022a57d4-a679-49c2-98ac-2c6a12178ff8-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.570138 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.570014    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-config\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.570427 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.570404    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/022a57d4-a679-49c2-98ac-2c6a12178ff8-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.573517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.573500    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/022a57d4-a679-49c2-98ac-2c6a12178ff8-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.612440 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.612380    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts25\" (UniqueName: \"kubernetes.io/projected/022a57d4-a679-49c2-98ac-2c6a12178ff8-kube-api-access-2ts25\") pod \"kube-storage-version-migrator-operator-79964ff6f9-4zcjv\" (UID: \"022a57d4-a679-49c2-98ac-2c6a12178ff8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.671103 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.671103 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671107    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r729\" (UniqueName: \"kubernetes.io/projected/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-kube-api-access-8r729\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.671336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671150    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-config\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.671336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671196    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcljc\" (UniqueName: \"kubernetes.io/projected/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-kube-api-access-wcljc\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.671336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qrd\" (UniqueName: \"kubernetes.io/projected/182b1f4c-e9ad-4de8-86a9-fddcadd3952f-kube-api-access-52qrd\") pod \"network-check-source-56499946bc-4bj29\" (UID: \"182b1f4c-e9ad-4de8-86a9-fddcadd3952f\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29"
Jul 01 20:47:11.671336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-config\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.671336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-trusted-ca\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.671573 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671358    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-serving-cert\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.671994 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.671970    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-config\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.672149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.672127    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-config\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.672355 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.672337    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-trusted-ca\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.673430 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.673401    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.673569 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.673551    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-serving-cert\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.689345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.689327    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcljc\" (UniqueName: \"kubernetes.io/projected/84f07d78-1d8e-48bf-abf6-3db7eae8ba51-kube-api-access-wcljc\") pod \"console-operator-7fdbdf9c99-g4bw7\" (UID: \"84f07d78-1d8e-48bf-abf6-3db7eae8ba51\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.690583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.690563    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r729\" (UniqueName: \"kubernetes.io/projected/0eed4a73-7e99-4236-b9ab-9c5de5b769c6-kube-api-access-8r729\") pod \"service-ca-operator-5d7b7dc95c-976gq\" (UID: \"0eed4a73-7e99-4236-b9ab-9c5de5b769c6\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.704295 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.704279    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:47:11.716772 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.716755    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:47:11.725326 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.725311    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"
Jul 01 20:47:11.725898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.725883    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qrd\" (UniqueName: \"kubernetes.io/projected/182b1f4c-e9ad-4de8-86a9-fddcadd3952f-kube-api-access-52qrd\") pod \"network-check-source-56499946bc-4bj29\" (UID: \"182b1f4c-e9ad-4de8-86a9-fddcadd3952f\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29"
Jul 01 20:47:11.831267 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.831235    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:11.837098 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.837075    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"
Jul 01 20:47:11.841584 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.841565    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29"
Jul 01 20:47:11.868525 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.868415    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv"]
Jul 01 20:47:11.872194 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:11.872162    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022a57d4_a679_49c2_98ac_2c6a12178ff8.slice/crio-ecec3db3ebd7d16f25be08fd4801da87901efc0873b1aecac58a7f80fd59c18d WatchSource:0}: Error finding container ecec3db3ebd7d16f25be08fd4801da87901efc0873b1aecac58a7f80fd59c18d: Status 404 returned error can't find the container with id ecec3db3ebd7d16f25be08fd4801da87901efc0873b1aecac58a7f80fd59c18d
Jul 01 20:47:11.976085 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.976054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:11.976271 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:11.976198    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:11.976271 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:11.976264    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:12.976247936 +0000 UTC m=+66.659731818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:11.989037 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:11.989005    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"]
Jul 01 20:47:11.992275 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:11.992249    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f07d78_1d8e_48bf_abf6_3db7eae8ba51.slice/crio-5f2341eb374b64ffcaa9b8913bcc55d0fee10352cbebabf6de92e68a9df4c2e1 WatchSource:0}: Error finding container 5f2341eb374b64ffcaa9b8913bcc55d0fee10352cbebabf6de92e68a9df4c2e1: Status 404 returned error can't find the container with id 5f2341eb374b64ffcaa9b8913bcc55d0fee10352cbebabf6de92e68a9df4c2e1
Jul 01 20:47:12.007338 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.007309    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" event={"ID":"84f07d78-1d8e-48bf-abf6-3db7eae8ba51","Type":"ContainerStarted","Data":"5f2341eb374b64ffcaa9b8913bcc55d0fee10352cbebabf6de92e68a9df4c2e1"}
Jul 01 20:47:12.008338 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.008316    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv" event={"ID":"022a57d4-a679-49c2-98ac-2c6a12178ff8","Type":"ContainerStarted","Data":"ecec3db3ebd7d16f25be08fd4801da87901efc0873b1aecac58a7f80fd59c18d"}
Jul 01 20:47:12.193134 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.193107    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq"]
Jul 01 20:47:12.195554 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:12.195525    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eed4a73_7e99_4236_b9ab_9c5de5b769c6.slice/crio-4878892aaab6434d288810ff6b494b52ed58927bbd0119bf01fe69c2c732a2d3 WatchSource:0}: Error finding container 4878892aaab6434d288810ff6b494b52ed58927bbd0119bf01fe69c2c732a2d3: Status 404 returned error can't find the container with id 4878892aaab6434d288810ff6b494b52ed58927bbd0119bf01fe69c2c732a2d3
Jul 01 20:47:12.197969 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.197947    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-4bj29"]
Jul 01 20:47:12.201263 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:12.201241    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182b1f4c_e9ad_4de8_86a9_fddcadd3952f.slice/crio-f54987b4d2b1eb426cc9363fef2a0079586eb68ef1313bf07dcd9e481b68f791 WatchSource:0}: Error finding container f54987b4d2b1eb426cc9363fef2a0079586eb68ef1313bf07dcd9e481b68f791: Status 404 returned error can't find the container with id f54987b4d2b1eb426cc9363fef2a0079586eb68ef1313bf07dcd9e481b68f791
Jul 01 20:47:12.582110 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.582017    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:47:12.582110 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.582082    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.582110    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.582156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.582380    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.582466    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls podName:f752519e-26b4-4130-b085-0b3b60b7a89e nodeName:}" failed. No retries permitted until 2026-07-01 20:47:44.582444588 +0000 UTC m=+98.265928483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls") pod "dns-default-jzns2" (UID: "f752519e-26b4-4130-b085-0b3b60b7a89e") : secret "dns-default-metrics-tls" not found
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.582629    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:47:12.582678 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.582647    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5579cbb788-phwkg: secret "image-registry-tls" not found
Jul 01 20:47:12.583009 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.582702    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls podName:571220a3-6342-421d-9e8d-53949a4b87ba nodeName:}" failed. No retries permitted until 2026-07-01 20:47:44.582688052 +0000 UTC m=+98.266171948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls") pod "image-registry-5579cbb788-phwkg" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba") : secret "image-registry-tls" not found
Jul 01 20:47:12.585220 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.585200    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:12.585333 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.585249    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:12.592675 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.592655    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 20:47:12.592834 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.592711    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs podName:8e9a83a3-4849-4baf-a9a7-781bf7974983 nodeName:}" failed. No retries permitted until 2026-07-01 20:48:16.592696322 +0000 UTC m=+130.276180216 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs") pod "network-metrics-daemon-7fwz5" (UID: "8e9a83a3-4849-4baf-a9a7-781bf7974983") : secret "metrics-daemon-secret" not found
Jul 01 20:47:12.600268 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.599891    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05f7d919-ddfa-438a-9dd8-e5c6d4ed8498-original-pull-secret\") pod \"global-pull-secret-syncer-cbm9x\" (UID: \"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498\") " pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:47:12.682720 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.682674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:47:12.685916 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.685865    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqt9c\" (UniqueName: \"kubernetes.io/projected/e4f34b82-ccc9-474f-8fd5-f332edbdacc6-kube-api-access-mqt9c\") pod \"network-check-target-9scdv\" (UID: \"e4f34b82-ccc9-474f-8fd5-f332edbdacc6\") " pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:47:12.885231 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.885168    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbm9x"
Jul 01 20:47:12.894419 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.894162    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tg27v\""
Jul 01 20:47:12.901695 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.901602    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:47:12.985798 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:12.985732    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:12.986529 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.986493    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:12.986601 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:12.986573    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.986551743 +0000 UTC m=+68.670035627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:13.017472 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:13.017386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq" event={"ID":"0eed4a73-7e99-4236-b9ab-9c5de5b769c6","Type":"ContainerStarted","Data":"4878892aaab6434d288810ff6b494b52ed58927bbd0119bf01fe69c2c732a2d3"}
Jul 01 20:47:13.018983 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:13.018906    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29" event={"ID":"182b1f4c-e9ad-4de8-86a9-fddcadd3952f","Type":"ContainerStarted","Data":"f54987b4d2b1eb426cc9363fef2a0079586eb68ef1313bf07dcd9e481b68f791"}
Jul 01 20:47:13.095616 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:13.095542    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbm9x"]
Jul 01 20:47:13.099880 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:13.099840    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f7d919_ddfa_438a_9dd8_e5c6d4ed8498.slice/crio-0a3af7c779bbca3bfe5111b12db790c77bc1e456e2c82330db226e0ed1108038 WatchSource:0}: Error finding container 0a3af7c779bbca3bfe5111b12db790c77bc1e456e2c82330db226e0ed1108038: Status 404 returned error can't find the container with id 0a3af7c779bbca3bfe5111b12db790c77bc1e456e2c82330db226e0ed1108038
Jul 01 20:47:13.104952 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:13.103767    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9scdv"]
Jul 01 20:47:13.106952 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:13.106894    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f34b82_ccc9_474f_8fd5_f332edbdacc6.slice/crio-e8b522f3086ce2bab8af32467c40c19c8a4c84be81ac43023d1ee7da4ddbcd12 WatchSource:0}: Error finding container e8b522f3086ce2bab8af32467c40c19c8a4c84be81ac43023d1ee7da4ddbcd12: Status 404 returned error can't find the container with id e8b522f3086ce2bab8af32467c40c19c8a4c84be81ac43023d1ee7da4ddbcd12
Jul 01 20:47:14.023256 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:14.023207    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbm9x" event={"ID":"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498","Type":"ContainerStarted","Data":"0a3af7c779bbca3bfe5111b12db790c77bc1e456e2c82330db226e0ed1108038"}
Jul 01 20:47:14.024556 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:14.024527    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9scdv" event={"ID":"e4f34b82-ccc9-474f-8fd5-f332edbdacc6","Type":"ContainerStarted","Data":"e8b522f3086ce2bab8af32467c40c19c8a4c84be81ac43023d1ee7da4ddbcd12"}
Jul 01 20:47:15.006146 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:15.006105    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:15.006421 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:15.006401    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:15.006514 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:15.006476    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:19.006455975 +0000 UTC m=+72.689939875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:15.999498 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:15.999239    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4dxpb"]
Jul 01 20:47:16.002577 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.002538    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.009172 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.009152    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 20:47:16.009297 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.009181    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 20:47:16.009297 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.009213    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 20:47:16.009297 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.009155    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 20:47:16.009981 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.009961    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vcwf4\""
Jul 01 20:47:16.027987 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.027874    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4dxpb"]
Jul 01 20:47:16.116066 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.116026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.116254 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.116078    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8htw\" (UniqueName: \"kubernetes.io/projected/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-api-access-d8htw\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.116254 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.116169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.116371 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.116297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7e8ef8f-02f7-4167-b375-4af7ec26e660-crio-socket\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.116371 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.116335    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e8ef8f-02f7-4167-b375-4af7ec26e660-data-volume\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217206 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217394 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217282    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7e8ef8f-02f7-4167-b375-4af7ec26e660-crio-socket\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217394 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e8ef8f-02f7-4167-b375-4af7ec26e660-data-volume\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217394 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217562 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217399    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8htw\" (UniqueName: \"kubernetes.io/projected/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-api-access-d8htw\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217754 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217710    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7e8ef8f-02f7-4167-b375-4af7ec26e660-crio-socket\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.217754 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.217742    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:16.217883 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.217754    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:47:16.217883 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.217817    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:16.717797737 +0000 UTC m=+70.401281635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : configmap "kube-rbac-proxy" not found
Jul 01 20:47:16.217883 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.217834    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:16.717824858 +0000 UTC m=+70.401308773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:16.218041 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.217982    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e8ef8f-02f7-4167-b375-4af7ec26e660-data-volume\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.239629 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.239603    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8htw\" (UniqueName: \"kubernetes.io/projected/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-api-access-d8htw\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.722252 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.722227    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.722360 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:16.722300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:16.722421 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.722357    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:47:16.722470 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.722423    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:17.722403709 +0000 UTC m=+71.405887607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : configmap "kube-rbac-proxy" not found
Jul 01 20:47:16.722470 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.722459    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:16.722581 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:16.722505    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:17.722490634 +0000 UTC m=+71.405974533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:17.038656 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.037970    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29" event={"ID":"182b1f4c-e9ad-4de8-86a9-fddcadd3952f","Type":"ContainerStarted","Data":"38c08557e55007d5795d57876c897f1ba79f0a017a8f336bb0206f8ba34ca7a3"}
Jul 01 20:47:17.040610 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.040584    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq" event={"ID":"0eed4a73-7e99-4236-b9ab-9c5de5b769c6","Type":"ContainerStarted","Data":"0f870b9682265e9d108f5162b0b460e854d32daf45200545d6cbe8dfb56ee307"}
Jul 01 20:47:17.043449 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.043421    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9scdv" event={"ID":"e4f34b82-ccc9-474f-8fd5-f332edbdacc6","Type":"ContainerStarted","Data":"6f1bc74da595894d746ea0a2a22b43bef4fc2b0223742c2ff6e3d8dd0add2c05"}
Jul 01 20:47:17.043605 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.043570    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:47:17.045984 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.045955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv" event={"ID":"022a57d4-a679-49c2-98ac-2c6a12178ff8","Type":"ContainerStarted","Data":"c152f41c4b7a4235cb69bc31b7fbf41031d44af2dbacab811c9855e31f400883"}
Jul 01 20:47:17.049023 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.049004    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/0.log"
Jul 01 20:47:17.049109 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.049044    2554 generic.go:356] "Generic (PLEG): container finished" podID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51" containerID="149792332ddf073684fa97ef70b5244fce91a0f5c8c6d6cfab9152bbcf18f15f" exitCode=255
Jul 01 20:47:17.049109 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.049071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" event={"ID":"84f07d78-1d8e-48bf-abf6-3db7eae8ba51","Type":"ContainerDied","Data":"149792332ddf073684fa97ef70b5244fce91a0f5c8c6d6cfab9152bbcf18f15f"}
Jul 01 20:47:17.049311 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.049295    2554 scope.go:117] "RemoveContainer" containerID="149792332ddf073684fa97ef70b5244fce91a0f5c8c6d6cfab9152bbcf18f15f"
Jul 01 20:47:17.067834 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.067677    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-4bj29" podStartSLOduration=1.5811480150000001 podStartE2EDuration="6.067659572s" totalImagesPullingTime="4.486511557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:17.065603106 +0000 UTC m=+70.749087049" watchObservedRunningTime="2026-07-01 20:47:17.067659572 +0000 UTC m=+70.751143473"
Jul 01 20:47:17.119300 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.118886    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9scdv" podStartSLOduration=66.502116251 podStartE2EDuration="1m10.118865224s" totalImagesPullingTime="3.616748973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:17.11806137 +0000 UTC m=+70.801545275" watchObservedRunningTime="2026-07-01 20:47:17.118865224 +0000 UTC m=+70.802349130"
Jul 01 20:47:17.141096 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.141038    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq" podStartSLOduration=1.654228261 podStartE2EDuration="6.141019573s" totalImagesPullingTime="4.486791312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:17.140776539 +0000 UTC m=+70.824260447" watchObservedRunningTime="2026-07-01 20:47:17.141019573 +0000 UTC m=+70.824503478"
Jul 01 20:47:17.170670 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.169982    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv" podStartSLOduration=1.368732077 podStartE2EDuration="6.169960401s" totalImagesPullingTime="4.801228324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:17.169190975 +0000 UTC m=+70.852674880" watchObservedRunningTime="2026-07-01 20:47:17.169960401 +0000 UTC m=+70.853444308"
Jul 01 20:47:17.731493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.731382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:17.731493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.731457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:17.731740 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:17.731505    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:47:17.731740 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:17.731570    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:19.731550404 +0000 UTC m=+73.415034290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : configmap "kube-rbac-proxy" not found
Jul 01 20:47:17.731740 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:17.731584    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:17.731740 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:17.731652    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:19.731633781 +0000 UTC m=+73.415117676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:17.792872 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:17.792843    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qn9x_78922f2f-7ca3-468e-ae75-5a9cb7ca1563/dns-node-resolver/0.log"
Jul 01 20:47:18.055175 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.055094    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:47:18.055696 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.055675    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/0.log"
Jul 01 20:47:18.055785 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.055724    2554 generic.go:356] "Generic (PLEG): container finished" podID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f" exitCode=255
Jul 01 20:47:18.055880 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.055857    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" event={"ID":"84f07d78-1d8e-48bf-abf6-3db7eae8ba51","Type":"ContainerDied","Data":"4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"}
Jul 01 20:47:18.055973 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.055910    2554 scope.go:117] "RemoveContainer" containerID="149792332ddf073684fa97ef70b5244fce91a0f5c8c6d6cfab9152bbcf18f15f"
Jul 01 20:47:18.056502 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.056482    2554 scope.go:117] "RemoveContainer" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"
Jul 01 20:47:18.056891 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:18.056638    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-g4bw7_openshift-console-operator(84f07d78-1d8e-48bf-abf6-3db7eae8ba51)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" podUID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51"
Jul 01 20:47:18.790325 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:18.790292    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24krb_153889c1-36ed-424f-ad2e-4e3d98b84921/node-ca/0.log"
Jul 01 20:47:19.048112 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.048046    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:19.048224 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.048183    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:19.048283 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.048262    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:27.048240464 +0000 UTC m=+80.731724347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:19.061363 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.061345    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:47:19.061839 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.061705    2554 scope.go:117] "RemoveContainer" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"
Jul 01 20:47:19.061916 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.061891    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-g4bw7_openshift-console-operator(84f07d78-1d8e-48bf-abf6-3db7eae8ba51)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" podUID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51"
Jul 01 20:47:19.590749 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.590721    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dq69_6632a983-a19e-47ef-a1d5-ff80bf434faf/serve-healthcheck-canary/0.log"
Jul 01 20:47:19.752571 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.752538    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:19.752746 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:19.752585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:19.752746 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.752672    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:47:19.752746 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.752687    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:19.752746 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.752738    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:23.752725058 +0000 UTC m=+77.436208940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:19.752914 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:19.752778    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:23.752767806 +0000 UTC m=+77.436251701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : configmap "kube-rbac-proxy" not found
Jul 01 20:47:20.066559 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:20.066522    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbm9x" event={"ID":"05f7d919-ddfa-438a-9dd8-e5c6d4ed8498","Type":"ContainerStarted","Data":"9aea5696159566e150ad46de08398b1c3d69b095a42915c619474ad21baf734c"}
Jul 01 20:47:20.097503 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:20.097440    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cbm9x" podStartSLOduration=67.186241793 podStartE2EDuration="1m13.097420662s" totalImagesPullingTime="5.911178869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.096565115 +0000 UTC m=+73.780049019" watchObservedRunningTime="2026-07-01 20:47:20.097420662 +0000 UTC m=+73.780904567"
Jul 01 20:47:21.831825 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:21.831780    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:21.832241 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:21.831857    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:21.832241 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:21.832181    2554 scope.go:117] "RemoveContainer" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"
Jul 01 20:47:21.832348 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:21.832328    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-g4bw7_openshift-console-operator(84f07d78-1d8e-48bf-abf6-3db7eae8ba51)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" podUID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51"
Jul 01 20:47:22.074205 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:22.074180    2554 scope.go:117] "RemoveContainer" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"
Jul 01 20:47:22.074379 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:22.074335    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-g4bw7_openshift-console-operator(84f07d78-1d8e-48bf-abf6-3db7eae8ba51)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" podUID="84f07d78-1d8e-48bf-abf6-3db7eae8ba51"
Jul 01 20:47:23.786312 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:23.786250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:23.786754 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:23.786401    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:23.786754 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:23.786423    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:23.786754 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:23.786496    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:31.786480739 +0000 UTC m=+85.469964632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:23.786754 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:23.786497    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:47:23.786754 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:23.786562    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:31.786544387 +0000 UTC m=+85.470028291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : configmap "kube-rbac-proxy" not found
Jul 01 20:47:27.113296 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:27.113256    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:27.113717 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:27.113405    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:47:27.113717 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:27.113466    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert podName:a2008ae7-08d6-4147-944b-492d48480330 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:43.113451111 +0000 UTC m=+96.796934997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-2rr8g" (UID: "a2008ae7-08d6-4147-944b-492d48480330") : secret "networking-console-plugin-cert" not found
Jul 01 20:47:31.854907 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:31.854785    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:31.854907 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:31.854840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:31.855311 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:31.854998    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:31.855311 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:31.855055    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls podName:e7e8ef8f-02f7-4167-b375-4af7ec26e660 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:47.855038248 +0000 UTC m=+101.538522149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dxpb" (UID: "e7e8ef8f-02f7-4167-b375-4af7ec26e660") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:31.855381 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:31.855349    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7e8ef8f-02f7-4167-b375-4af7ec26e660-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:35.767376 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:35.767340    2554 scope.go:117] "RemoveContainer" containerID="4017542a523bf09ab5cfece45304b802496134ce8af59167fbf6128f66cf2b3f"
Jul 01 20:47:36.118822 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:36.118750    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:47:36.118822 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:36.118819    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" event={"ID":"84f07d78-1d8e-48bf-abf6-3db7eae8ba51","Type":"ContainerStarted","Data":"e73d375df06244387df1f9351676eba99ab024734c53f0da7f7ca90161e69c07"}
Jul 01 20:47:36.119154 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:36.119132    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:36.150090 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:36.150044    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7" podStartSLOduration=20.468294998 podStartE2EDuration="25.150031794s" totalImagesPullingTime="4.681736796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:36.148153934 +0000 UTC m=+89.831637838" watchObservedRunningTime="2026-07-01 20:47:36.150031794 +0000 UTC m=+89.833515697"
Jul 01 20:47:36.531786 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:36.531758    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-g4bw7"
Jul 01 20:47:43.138783 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.138748    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:43.141096 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.141073    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2008ae7-08d6-4147-944b-492d48480330-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2rr8g\" (UID: \"a2008ae7-08d6-4147-944b-492d48480330\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:43.155929 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.155912    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"
Jul 01 20:47:43.210083 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.209466    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"]
Jul 01 20:47:43.213061 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.212663    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.218622 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.218288    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 20:47:43.218622 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.218522    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 20:47:43.218807 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.218643    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 20:47:43.221318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.221049    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 20:47:43.237278 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.236901    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"]
Jul 01 20:47:43.304671 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.304645    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g"]
Jul 01 20:47:43.307269 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:43.307245    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2008ae7_08d6_4147_944b_492d48480330.slice/crio-b97513502d1cb62365bddf33b699c1c34bb61bc0833872e27dcff6e9694cef27 WatchSource:0}: Error finding container b97513502d1cb62365bddf33b699c1c34bb61bc0833872e27dcff6e9694cef27: Status 404 returned error can't find the container with id b97513502d1cb62365bddf33b699c1c34bb61bc0833872e27dcff6e9694cef27
Jul 01 20:47:43.340431 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.340410    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad3f81d9-7cc5-4293-a27a-321b7409dea5-klusterlet-config\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.340531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.340442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3f81d9-7cc5-4293-a27a-321b7409dea5-tmp\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.340531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.340480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4c82\" (UniqueName: \"kubernetes.io/projected/ad3f81d9-7cc5-4293-a27a-321b7409dea5-kube-api-access-r4c82\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.441177 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.441109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4c82\" (UniqueName: \"kubernetes.io/projected/ad3f81d9-7cc5-4293-a27a-321b7409dea5-kube-api-access-r4c82\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.441287 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.441221    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad3f81d9-7cc5-4293-a27a-321b7409dea5-klusterlet-config\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.441287 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.441255    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3f81d9-7cc5-4293-a27a-321b7409dea5-tmp\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.441624 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.441600    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3f81d9-7cc5-4293-a27a-321b7409dea5-tmp\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.443614 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.443594    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad3f81d9-7cc5-4293-a27a-321b7409dea5-klusterlet-config\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.459614 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.459595    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4c82\" (UniqueName: \"kubernetes.io/projected/ad3f81d9-7cc5-4293-a27a-321b7409dea5-kube-api-access-r4c82\") pod \"klusterlet-addon-workmgr-779cfbc978-9xlgd\" (UID: \"ad3f81d9-7cc5-4293-a27a-321b7409dea5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.525920 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.525892    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:43.668220 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:43.668145    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"]
Jul 01 20:47:43.671400 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:43.671376    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3f81d9_7cc5_4293_a27a_321b7409dea5.slice/crio-dbc6a38111fb92b3c027ece60991c1ee7643b7cda3c3a847c57ec8460e1b4e54 WatchSource:0}: Error finding container dbc6a38111fb92b3c027ece60991c1ee7643b7cda3c3a847c57ec8460e1b4e54: Status 404 returned error can't find the container with id dbc6a38111fb92b3c027ece60991c1ee7643b7cda3c3a847c57ec8460e1b4e54
Jul 01 20:47:44.143324 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.143287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g" event={"ID":"a2008ae7-08d6-4147-944b-492d48480330","Type":"ContainerStarted","Data":"b97513502d1cb62365bddf33b699c1c34bb61bc0833872e27dcff6e9694cef27"}
Jul 01 20:47:44.144612 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.144582    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd" event={"ID":"ad3f81d9-7cc5-4293-a27a-321b7409dea5","Type":"ContainerStarted","Data":"dbc6a38111fb92b3c027ece60991c1ee7643b7cda3c3a847c57ec8460e1b4e54"}
Jul 01 20:47:44.651209 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.651121    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:44.651209 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.651159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:47:44.653683 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.653658    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f752519e-26b4-4130-b085-0b3b60b7a89e-metrics-tls\") pod \"dns-default-jzns2\" (UID: \"f752519e-26b4-4130-b085-0b3b60b7a89e\") " pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:44.654045 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.654024    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"image-registry-5579cbb788-phwkg\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") " pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:47:44.899272 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.899236    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgzdr\""
Jul 01 20:47:44.907136 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.907061    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:47:44.915556 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.915533    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4r2bh\""
Jul 01 20:47:44.923727 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:44.923527    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:45.085927 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.085855    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:47:45.089331 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:45.089303    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571220a3_6342_421d_9e8d_53949a4b87ba.slice/crio-863290b975659df333b161b01bf1d343748f2d4acbb7d458beebc62163fdbb89 WatchSource:0}: Error finding container 863290b975659df333b161b01bf1d343748f2d4acbb7d458beebc62163fdbb89: Status 404 returned error can't find the container with id 863290b975659df333b161b01bf1d343748f2d4acbb7d458beebc62163fdbb89
Jul 01 20:47:45.094561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.094542    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzns2"]
Jul 01 20:47:45.098086 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:45.098028    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf752519e_26b4_4130_b085_0b3b60b7a89e.slice/crio-78738d18f39efa1b75275d7f9e9527f4f4681e3f370fb6ed91b8cfdfec4ae44f WatchSource:0}: Error finding container 78738d18f39efa1b75275d7f9e9527f4f4681e3f370fb6ed91b8cfdfec4ae44f: Status 404 returned error can't find the container with id 78738d18f39efa1b75275d7f9e9527f4f4681e3f370fb6ed91b8cfdfec4ae44f
Jul 01 20:47:45.150677 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.150608    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzns2" event={"ID":"f752519e-26b4-4130-b085-0b3b60b7a89e","Type":"ContainerStarted","Data":"78738d18f39efa1b75275d7f9e9527f4f4681e3f370fb6ed91b8cfdfec4ae44f"}
Jul 01 20:47:45.152552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.152525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" event={"ID":"571220a3-6342-421d-9e8d-53949a4b87ba","Type":"ContainerStarted","Data":"863290b975659df333b161b01bf1d343748f2d4acbb7d458beebc62163fdbb89"}
Jul 01 20:47:45.154509 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.154302    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g" event={"ID":"a2008ae7-08d6-4147-944b-492d48480330","Type":"ContainerStarted","Data":"0138eb238811ffec314bd75c70434e2224047e0692b18dbbc416ce3e8beb6413"}
Jul 01 20:47:45.179993 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.179847    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2rr8g" podStartSLOduration=33.106575112 podStartE2EDuration="34.179830475s" totalImagesPullingTime="1.073255363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:45.178671923 +0000 UTC m=+98.862155829" watchObservedRunningTime="2026-07-01 20:47:45.179830475 +0000 UTC m=+98.863314380"
Jul 01 20:47:45.300743 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.300706    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:47:45.304793 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.304105    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.310152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.309665    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 20:47:45.310152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.309965    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 20:47:45.310152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.310028    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 20:47:45.313346 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.313142    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 20:47:45.313346 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.313182    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-86w5c\""
Jul 01 20:47:45.313517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.313499    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 20:47:45.314244 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.314149    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 20:47:45.314820 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.314624    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 20:47:45.321714 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.321657    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:47:45.459413 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.459413 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459368    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.459413 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.459728 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsv6m\" (UniqueName: \"kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.459728 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459467    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.459728 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.459595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560388    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560422    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.560499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsv6m\" (UniqueName: \"kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.561072    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.561461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.561385    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.562192 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.562020    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.564566 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.564503    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.567596 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.567148    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.573814 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.573777    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsv6m\" (UniqueName: \"kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m\") pod \"console-85b845f874-nb8dw\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") " pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:45.618859 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:45.618603    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:46.160240 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.160197    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" event={"ID":"571220a3-6342-421d-9e8d-53949a4b87ba","Type":"ContainerStarted","Data":"6d9cb702009f8b8c7c081691e2fe70152681dcc402b23c84f4d440b47d5e994d"}
Jul 01 20:47:46.160692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.160282    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:47:46.179763 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.179735    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"]
Jul 01 20:47:46.192728 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.192669    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" podStartSLOduration=98.192649271 podStartE2EDuration="1m38.192649271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:46.190532049 +0000 UTC m=+99.874015955" watchObservedRunningTime="2026-07-01 20:47:46.192649271 +0000 UTC m=+99.876133176"
Jul 01 20:47:46.264038 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.264003    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"]
Jul 01 20:47:46.264259 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.264132    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:46.267502 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.267235    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 01 20:47:46.267502 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.267349    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hfvl5\""
Jul 01 20:47:46.367440 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.367397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/598eb134-3f95-4bac-b5df-6242a4746e38-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-kwzdl\" (UID: \"598eb134-3f95-4bac-b5df-6242a4746e38\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:46.468111 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.468037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/598eb134-3f95-4bac-b5df-6242a4746e38-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-kwzdl\" (UID: \"598eb134-3f95-4bac-b5df-6242a4746e38\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:46.470351 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.470320    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/598eb134-3f95-4bac-b5df-6242a4746e38-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-kwzdl\" (UID: \"598eb134-3f95-4bac-b5df-6242a4746e38\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:46.576406 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:46.576363    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:47.879660 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:47.879613    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:47.882031 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:47.882006    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7e8ef8f-02f7-4167-b375-4af7ec26e660-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dxpb\" (UID: \"e7e8ef8f-02f7-4167-b375-4af7ec26e660\") " pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:48.059690 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:48.059654    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9scdv"
Jul 01 20:47:48.115144 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:48.115116    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4dxpb"
Jul 01 20:47:48.960089 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:48.959339    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"]
Jul 01 20:47:48.966764 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:48.966730    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598eb134_3f95_4bac_b5df_6242a4746e38.slice/crio-c5b1546139b4f63a73b59fa6efb1ac916dba0d9a1f31f8ddb9e401ed1682d4f1 WatchSource:0}: Error finding container c5b1546139b4f63a73b59fa6efb1ac916dba0d9a1f31f8ddb9e401ed1682d4f1: Status 404 returned error can't find the container with id c5b1546139b4f63a73b59fa6efb1ac916dba0d9a1f31f8ddb9e401ed1682d4f1
Jul 01 20:47:48.984844 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:48.984775    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:47:49.000755 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.000729    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4dxpb"]
Jul 01 20:47:49.176898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.176865    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl" event={"ID":"598eb134-3f95-4bac-b5df-6242a4746e38","Type":"ContainerStarted","Data":"c5b1546139b4f63a73b59fa6efb1ac916dba0d9a1f31f8ddb9e401ed1682d4f1"}
Jul 01 20:47:49.178101 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.178068    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dxpb" event={"ID":"e7e8ef8f-02f7-4167-b375-4af7ec26e660","Type":"ContainerStarted","Data":"392c8dbb148b4ef8685b212ea8bdd24b373e147240240b39e7d4fade5b4eae50"}
Jul 01 20:47:49.179308 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.179287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b845f874-nb8dw" event={"ID":"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7","Type":"ContainerStarted","Data":"f19ed4ad2d2a73b779333554a0c97e49766eb30461c11c36de9334603f11ecf9"}
Jul 01 20:47:49.180660 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.180637    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd" event={"ID":"ad3f81d9-7cc5-4293-a27a-321b7409dea5","Type":"ContainerStarted","Data":"a237607ede1e90205ae63715ac19a5c5623ad7fe6fb86e06f10936421f7272fa"}
Jul 01 20:47:49.180805 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.180780    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:49.182504 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.182483    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzns2" event={"ID":"f752519e-26b4-4130-b085-0b3b60b7a89e","Type":"ContainerStarted","Data":"b2b2ed2062f10c42c8d09b24327ddbc559b3faa6a9b9be47d87507b6e160a9fd"}
Jul 01 20:47:49.182594 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.182526    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzns2" event={"ID":"f752519e-26b4-4130-b085-0b3b60b7a89e","Type":"ContainerStarted","Data":"feae65f7239e3bc59f6836b04e7dd8af6d20852881c436bdbecfff6e0f5efffa"}
Jul 01 20:47:49.182594 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.182564    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:49.182663 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.182614    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd"
Jul 01 20:47:49.198037 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.197994    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-779cfbc978-9xlgd" podStartSLOduration=1.089463124 podStartE2EDuration="6.197982385s" totalImagesPullingTime="5.108519261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:49.197557736 +0000 UTC m=+102.881041861" watchObservedRunningTime="2026-07-01 20:47:49.197982385 +0000 UTC m=+102.881466289"
Jul 01 20:47:49.217191 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:49.217151    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jzns2" podStartSLOduration=65.533798653 podStartE2EDuration="1m9.21713851s" totalImagesPullingTime="3.683339857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:49.216446729 +0000 UTC m=+102.899930644" watchObservedRunningTime="2026-07-01 20:47:49.21713851 +0000 UTC m=+102.900622418"
Jul 01 20:47:51.192633 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:51.192587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl" event={"ID":"598eb134-3f95-4bac-b5df-6242a4746e38","Type":"ContainerStarted","Data":"e1cc05d09954005dfa8d68f53fb2ae82e476475d34af28f7c06d67d227ff7be1"}
Jul 01 20:47:51.211693 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:51.211643    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl" podStartSLOduration=3.4856031610000002 podStartE2EDuration="5.211623637s" totalImagesPullingTime="1.726020476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:51.210001725 +0000 UTC m=+104.893485677" watchObservedRunningTime="2026-07-01 20:47:51.211623637 +0000 UTC m=+104.895107543"
Jul 01 20:47:52.196199 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:52.196161    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:52.201898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:52.201871    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-kwzdl"
Jul 01 20:47:53.269173 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.269143    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-2gssg"]
Jul 01 20:47:53.277326 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.277302    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.280064 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.280032    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 20:47:53.280289 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.280270    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 20:47:53.280340 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.280289    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 20:47:53.281550 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.281517    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 20:47:53.281662 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.281563    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 20:47:53.281736 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.281677    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sq58f\""
Jul 01 20:47:53.284075 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.284040    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-2gssg"]
Jul 01 20:47:53.429359 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.429328    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.429550 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.429406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.429550 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.429435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.429550 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.429464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbnm\" (UniqueName: \"kubernetes.io/projected/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-kube-api-access-nvbnm\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.530124 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.530050    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.530258 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.530149    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.530258 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.530185    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.530258 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.530216    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbnm\" (UniqueName: \"kubernetes.io/projected/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-kube-api-access-nvbnm\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.530396 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:53.530288    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 01 20:47:53.530396 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:53.530347    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls podName:01dcc36e-9a6d-4cec-b76d-3e9006849b7a nodeName:}" failed. No retries permitted until 2026-07-01 20:47:54.030333389 +0000 UTC m=+107.713817275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-2gssg" (UID: "01dcc36e-9a6d-4cec-b76d-3e9006849b7a") : secret "prometheus-operator-tls" not found
Jul 01 20:47:53.530770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.530751    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.532497 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.532471    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:53.539788 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:53.539769    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbnm\" (UniqueName: \"kubernetes.io/projected/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-kube-api-access-nvbnm\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:54.033466 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:54.033431    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:54.035797 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:54.035772    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01dcc36e-9a6d-4cec-b76d-3e9006849b7a-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-2gssg\" (UID: \"01dcc36e-9a6d-4cec-b76d-3e9006849b7a\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:54.188965 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:54.188911    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg"
Jul 01 20:47:54.544893 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:54.544694    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-2gssg"]
Jul 01 20:47:54.547438 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:54.547410    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dcc36e_9a6d_4cec_b76d_3e9006849b7a.slice/crio-3bdf0a47525e298ec64a08dd09cb0d002bfa0094f2f2eb227e1b9b761f6e192c WatchSource:0}: Error finding container 3bdf0a47525e298ec64a08dd09cb0d002bfa0094f2f2eb227e1b9b761f6e192c: Status 404 returned error can't find the container with id 3bdf0a47525e298ec64a08dd09cb0d002bfa0094f2f2eb227e1b9b761f6e192c
Jul 01 20:47:55.208369 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.208344    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dxpb" event={"ID":"e7e8ef8f-02f7-4167-b375-4af7ec26e660","Type":"ContainerStarted","Data":"c492bd388f89e2d51dd3abced59b6255bd1f2e7d11b423aa79f823169d9fff67"}
Jul 01 20:47:55.208463 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.208401    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dxpb" event={"ID":"e7e8ef8f-02f7-4167-b375-4af7ec26e660","Type":"ContainerStarted","Data":"68364cc9a6fa73a6ac5447a62593fec28a89f4cc05f38231e3d18c032c49e994"}
Jul 01 20:47:55.209648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.209625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b845f874-nb8dw" event={"ID":"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7","Type":"ContainerStarted","Data":"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"}
Jul 01 20:47:55.210677 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.210657    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg" event={"ID":"01dcc36e-9a6d-4cec-b76d-3e9006849b7a","Type":"ContainerStarted","Data":"3bdf0a47525e298ec64a08dd09cb0d002bfa0094f2f2eb227e1b9b761f6e192c"}
Jul 01 20:47:55.231200 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.231154    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85b845f874-nb8dw" podStartSLOduration=4.807038769 podStartE2EDuration="10.23113701s" totalImagesPullingTime="5.424098241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:55.230585248 +0000 UTC m=+108.914069152" watchObservedRunningTime="2026-07-01 20:47:55.23113701 +0000 UTC m=+108.914620913"
Jul 01 20:47:55.619884 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.619838    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:55.620333 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.619969    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:55.626098 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:55.626070    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:56.217275 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:56.217233    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg" event={"ID":"01dcc36e-9a6d-4cec-b76d-3e9006849b7a","Type":"ContainerStarted","Data":"c34c59507bbb995bae22b554a6b7ea0e23390db1311fa8ed5ac7211316319291"}
Jul 01 20:47:56.217461 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:56.217341    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg" event={"ID":"01dcc36e-9a6d-4cec-b76d-3e9006849b7a","Type":"ContainerStarted","Data":"5191cb472742366f04c63c44ef471cee5549d9aaa706e258c94d6c16c0f39263"}
Jul 01 20:47:56.221959 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:56.221916    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:47:56.238508 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:56.238434    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-2gssg" podStartSLOduration=2.027523138 podStartE2EDuration="3.238414013s" totalImagesPullingTime="1.210890875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:56.237376787 +0000 UTC m=+109.920860692" watchObservedRunningTime="2026-07-01 20:47:56.238414013 +0000 UTC m=+109.921897918"
Jul 01 20:47:58.227038 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.226951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dxpb" event={"ID":"e7e8ef8f-02f7-4167-b375-4af7ec26e660","Type":"ContainerStarted","Data":"c361a67ade3e65f5aa9b888a0ecc6103de2b30c5c9f645805a6fa7eda6fa2c2c"}
Jul 01 20:47:58.250602 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.250432    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4dxpb" podStartSLOduration=34.476991857 podStartE2EDuration="43.25041768s" totalImagesPullingTime="8.773425823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:58.249971615 +0000 UTC m=+111.933455576" watchObservedRunningTime="2026-07-01 20:47:58.25041768 +0000 UTC m=+111.933901583"
Jul 01 20:47:58.973327 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.973291    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9hxdh"]
Jul 01 20:47:58.976711 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.976690    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:58.979789 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.979771    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzbn8\""
Jul 01 20:47:58.980106 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.980089    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 20:47:58.980194 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.980096    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 20:47:58.980374 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:58.980360    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 20:47:59.175181 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-textfile\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9kc\" (UniqueName: \"kubernetes.io/projected/15a785b1-a9ad-42d3-b435-895abbfd3b3e-kube-api-access-pg9kc\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175243    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-metrics-client-ca\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175312    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-root\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-sys\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175576 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175353    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.175576 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.175403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-wtmp\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.189493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.189469    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jzns2"
Jul 01 20:47:59.276481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.276481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276439    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-metrics-client-ca\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.276481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.276481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-root\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276490    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-sys\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276506    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-wtmp\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276567    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-root\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276576    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-textfile\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276575    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-sys\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:59.276657    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276693    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-wtmp\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9kc\" (UniqueName: \"kubernetes.io/projected/15a785b1-a9ad-42d3-b435-895abbfd3b3e-kube-api-access-pg9kc\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:47:59.276718    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls podName:15a785b1-a9ad-42d3-b435-895abbfd3b3e nodeName:}" failed. No retries permitted until 2026-07-01 20:47:59.776697819 +0000 UTC m=+113.460181714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls") pod "node-exporter-9hxdh" (UID: "15a785b1-a9ad-42d3-b435-895abbfd3b3e") : secret "node-exporter-tls" not found
Jul 01 20:47:59.277097 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.276991    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-textfile\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277465 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.277127    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.277465 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.277153    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a785b1-a9ad-42d3-b435-895abbfd3b3e-metrics-client-ca\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.278735 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.278716    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.294458 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.294433    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9kc\" (UniqueName: \"kubernetes.io/projected/15a785b1-a9ad-42d3-b435-895abbfd3b3e-kube-api-access-pg9kc\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.780724 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.780688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.783045 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.783013    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15a785b1-a9ad-42d3-b435-895abbfd3b3e-node-exporter-tls\") pod \"node-exporter-9hxdh\" (UID: \"15a785b1-a9ad-42d3-b435-895abbfd3b3e\") " pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.885736 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:47:59.885702    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hxdh"
Jul 01 20:47:59.898440 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:47:59.898411    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a785b1_a9ad_42d3_b435_895abbfd3b3e.slice/crio-1e78b385e4e4e2a6cc8926f987879fcd601dea3ec10a5a1e185ca292efd2c973 WatchSource:0}: Error finding container 1e78b385e4e4e2a6cc8926f987879fcd601dea3ec10a5a1e185ca292efd2c973: Status 404 returned error can't find the container with id 1e78b385e4e4e2a6cc8926f987879fcd601dea3ec10a5a1e185ca292efd2c973
Jul 01 20:48:00.015066 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.015037    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:48:00.019691 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.019674    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.027105 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.027087    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 20:48:00.027303 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.027291    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k4nff\""
Jul 01 20:48:00.028757 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.028744    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 20:48:00.029330 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.029317    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 20:48:00.030079 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.030062    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 20:48:00.030141 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.030063    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 20:48:00.030141 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.030063    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 20:48:00.031775 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.031730    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 20:48:00.032011 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.031996    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 20:48:00.033390 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.033372    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 20:48:00.066144 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.066124    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:48:00.083313 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2c9j\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-kube-api-access-b2c9j\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083417 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083332    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083417 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083382    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083417 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083407    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083567 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083567 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-web-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083567 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083510    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-config-out\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083567 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083545    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083752 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083575    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083752 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083752 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083752 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083669    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.083752 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.083690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-web-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184456    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-config-out\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184491    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184634 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184592    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184867 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184729    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184867 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184763    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2c9j\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-kube-api-access-b2c9j\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.184867 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184817    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.185052 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184882    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.185052 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184910    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.185052 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.184967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.185052 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.185030    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.186419 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.185681    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.186419 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.186285    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86f04a3-572c-4a38-bfae-573a0d01835f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.187531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.187499    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b86f04a3-572c-4a38-bfae-573a0d01835f-config-out\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.187681 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.187535    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.187681 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.187658    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-web-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.187801 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.187762    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.188338 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.188316    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.188439 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.188359    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.188683 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.188661    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.189077 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.189057    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.189541 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.189525    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b86f04a3-572c-4a38-bfae-573a0d01835f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.217013 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.216991    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2c9j\" (UniqueName: \"kubernetes.io/projected/b86f04a3-572c-4a38-bfae-573a0d01835f-kube-api-access-b2c9j\") pod \"alertmanager-main-0\" (UID: \"b86f04a3-572c-4a38-bfae-573a0d01835f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.233748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.233726    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hxdh" event={"ID":"15a785b1-a9ad-42d3-b435-895abbfd3b3e","Type":"ContainerStarted","Data":"1e78b385e4e4e2a6cc8926f987879fcd601dea3ec10a5a1e185ca292efd2c973"}
Jul 01 20:48:00.328857 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.328775    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:48:00.521915 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:00.521884    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:48:00.694104 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:48:00.694064    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86f04a3_572c_4a38_bfae_573a0d01835f.slice/crio-58c05d33f73d928d0990b2a632e00d2aae03e4d9dd6ba2ecea2ac0828c2a5e99 WatchSource:0}: Error finding container 58c05d33f73d928d0990b2a632e00d2aae03e4d9dd6ba2ecea2ac0828c2a5e99: Status 404 returned error can't find the container with id 58c05d33f73d928d0990b2a632e00d2aae03e4d9dd6ba2ecea2ac0828c2a5e99
Jul 01 20:48:01.239846 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:01.239802    2554 generic.go:356] "Generic (PLEG): container finished" podID="15a785b1-a9ad-42d3-b435-895abbfd3b3e" containerID="92c0a0c60eb117591a283d3ad212385279d400039267ff2bf7e581aa14b1e7c6" exitCode=0
Jul 01 20:48:01.240041 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:01.239894    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hxdh" event={"ID":"15a785b1-a9ad-42d3-b435-895abbfd3b3e","Type":"ContainerDied","Data":"92c0a0c60eb117591a283d3ad212385279d400039267ff2bf7e581aa14b1e7c6"}
Jul 01 20:48:01.241481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:01.241447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"58c05d33f73d928d0990b2a632e00d2aae03e4d9dd6ba2ecea2ac0828c2a5e99"}
Jul 01 20:48:02.246767 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:02.246733    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hxdh" event={"ID":"15a785b1-a9ad-42d3-b435-895abbfd3b3e","Type":"ContainerStarted","Data":"8657dcf84289d5be3940f6364e1a10a7aff2e774fd803d0317805c8ed4ce5b19"}
Jul 01 20:48:02.247418 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:02.246796    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hxdh" event={"ID":"15a785b1-a9ad-42d3-b435-895abbfd3b3e","Type":"ContainerStarted","Data":"e3d158afb848199ac2aa53369a421f39e4a09f8179e39d3eb0279b84d2ed42ed"}
Jul 01 20:48:02.248193 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:02.248165    2554 generic.go:356] "Generic (PLEG): container finished" podID="b86f04a3-572c-4a38-bfae-573a0d01835f" containerID="d33030302e8349cbf70163999ef8a45b3c1cd970774a578b8f9998ce5cc6f904" exitCode=0
Jul 01 20:48:02.248281 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:02.248246    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerDied","Data":"d33030302e8349cbf70163999ef8a45b3c1cd970774a578b8f9998ce5cc6f904"}
Jul 01 20:48:02.295278 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:02.295235    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9hxdh" podStartSLOduration=3.392391984 podStartE2EDuration="4.295218949s" totalImagesPullingTime="840.270943ms" totalInitContainerRuntime="62.556022ms" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:02.295181288 +0000 UTC m=+115.978665193" watchObservedRunningTime="2026-07-01 20:48:02.295218949 +0000 UTC m=+115.978702847"
Jul 01 20:48:03.675689 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.675652    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"]
Jul 01 20:48:03.680169 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.680149    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.684399 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.684378    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cpx4h\""
Jul 01 20:48:03.688744 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.688713    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 20:48:03.706706 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.706681    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"]
Jul 01 20:48:03.721635 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.721612    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c78cb989c-7z9vp"]
Jul 01 20:48:03.725180 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.725160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.727755 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.727730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.727914 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.727885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/cfee7caf-006b-47f1-a417-274f5a11abfb-kube-api-access-pvqz6\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.732947 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.732907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 20:48:03.733103 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.733081    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dxvw8\""
Jul 01 20:48:03.733482 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.733464    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 20:48:03.733799 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.733779    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 20:48:03.733909 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.733816    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 20:48:03.735425 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.735402    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1l3kv1vhiftog\""
Jul 01 20:48:03.760514 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.760487    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c78cb989c-7z9vp"]
Jul 01 20:48:03.828783 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.828747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.828965 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.828832    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/cfee7caf-006b-47f1-a417-274f5a11abfb-kube-api-access-pvqz6\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.829108 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-client-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829177 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829160    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-client-certs\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829191    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-metrics-server-audit-profiles\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829256 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829228    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.829286 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829269    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-audit-log\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829328 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829312    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrf8\" (UniqueName: \"kubernetes.io/projected/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-kube-api-access-6rrf8\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829385 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:48:03.829336    2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 01 20:48:03.829436 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.829381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-tls\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.829436 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:48:03.829400    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert podName:cfee7caf-006b-47f1-a417-274f5a11abfb nodeName:}" failed. No retries permitted until 2026-07-01 20:48:04.329380094 +0000 UTC m=+118.012863993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert") pod "monitoring-plugin-cd8656b8-9fbqd" (UID: "cfee7caf-006b-47f1-a417-274f5a11abfb") : secret "monitoring-plugin-cert" not found
Jul 01 20:48:03.848648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.848572    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/cfee7caf-006b-47f1-a417-274f5a11abfb-kube-api-access-pvqz6\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:03.930771 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.930744    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-audit-log\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.930851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.930796    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrf8\" (UniqueName: \"kubernetes.io/projected/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-kube-api-access-6rrf8\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.930851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.930836    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-tls\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.930986 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.930866    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.931851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.931152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-client-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.931851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.931174    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-audit-log\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.931851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.931229    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-client-certs\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.931851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.931258    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-metrics-server-audit-profiles\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.931851 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.931539    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.932201 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.932181    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-metrics-server-audit-profiles\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.933588 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.933545    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-tls\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.933903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.933882    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-secret-metrics-server-client-certs\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.934166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.934148    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-client-ca-bundle\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:03.954891 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:03.954872    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrf8\" (UniqueName: \"kubernetes.io/projected/31a7f6cb-d4c0-4cbc-adeb-ea37c920adae-kube-api-access-6rrf8\") pod \"metrics-server-5c78cb989c-7z9vp\" (UID: \"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae\") " pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:04.035224 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.035205    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:04.207889 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.207862    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c78cb989c-7z9vp"]
Jul 01 20:48:04.211006 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:48:04.210976    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a7f6cb_d4c0_4cbc_adeb_ea37c920adae.slice/crio-3a1e844736cf1ae35152c97b4a551e4128e7dd05f2e3a91bb21d99e5f6a86b01 WatchSource:0}: Error finding container 3a1e844736cf1ae35152c97b4a551e4128e7dd05f2e3a91bb21d99e5f6a86b01: Status 404 returned error can't find the container with id 3a1e844736cf1ae35152c97b4a551e4128e7dd05f2e3a91bb21d99e5f6a86b01
Jul 01 20:48:04.256763 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.256730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp" event={"ID":"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae","Type":"ContainerStarted","Data":"3a1e844736cf1ae35152c97b4a551e4128e7dd05f2e3a91bb21d99e5f6a86b01"}
Jul 01 20:48:04.259581 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.259556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"80c2bb1aed311f9f77f7099a4edda093dd9341b13b61debfa68e036e2d06146c"}
Jul 01 20:48:04.259692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.259620    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"ab783e471434d9f86aeec98b014f340481eab9a8b7d5d0d930d9bfaadfa44959"}
Jul 01 20:48:04.259692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.259647    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"4d1926582bf0e5d8ca72af27947934e3e1409922e17f89b33e6eb6ae7e670e44"}
Jul 01 20:48:04.259692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.259671    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"56d1d31f6eae036a2f01a06b2e7276b75bdf5f9c9b415df49ed000bbab90052f"}
Jul 01 20:48:04.259805 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.259693    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"daf6af680cb4d4dd33e08a83e47c09995ac87edfb86297842f20105060fc34a5"}
Jul 01 20:48:04.334924 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.334885    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:04.337144 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.337124    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfee7caf-006b-47f1-a417-274f5a11abfb-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-9fbqd\" (UID: \"cfee7caf-006b-47f1-a417-274f5a11abfb\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:04.590997 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.590901    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:04.742871 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:04.742827    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"]
Jul 01 20:48:04.802036 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:48:04.802008    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfee7caf_006b_47f1_a417_274f5a11abfb.slice/crio-3c51b50a24bc01eff61833ded22b5ef16590057f17eb93435ea44f0f1b4533d0 WatchSource:0}: Error finding container 3c51b50a24bc01eff61833ded22b5ef16590057f17eb93435ea44f0f1b4533d0: Status 404 returned error can't find the container with id 3c51b50a24bc01eff61833ded22b5ef16590057f17eb93435ea44f0f1b4533d0
Jul 01 20:48:05.211915 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:05.211880    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:48:05.217050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:05.217027    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:48:05.269472 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:05.269438    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b86f04a3-572c-4a38-bfae-573a0d01835f","Type":"ContainerStarted","Data":"c7ae40aa9da237351e0ccf0ffcc2075ae702198da6ab64e26786fc31a508c1c7"}
Jul 01 20:48:05.271093 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:05.271061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd" event={"ID":"cfee7caf-006b-47f1-a417-274f5a11abfb","Type":"ContainerStarted","Data":"3c51b50a24bc01eff61833ded22b5ef16590057f17eb93435ea44f0f1b4533d0"}
Jul 01 20:48:05.405849 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:05.405796    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.085378274 podStartE2EDuration="6.405778914s" totalImagesPullingTime="3.300424729s" totalInitContainerRuntime="19.975911ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:05.404971753 +0000 UTC m=+119.088455657" watchObservedRunningTime="2026-07-01 20:48:05.405778914 +0000 UTC m=+119.089262874"
Jul 01 20:48:07.280583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.280542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd" event={"ID":"cfee7caf-006b-47f1-a417-274f5a11abfb","Type":"ContainerStarted","Data":"7225202dba20189a57d3f232384df9115e60ab7adc80349695dd9417a4915a48"}
Jul 01 20:48:07.281037 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.280722    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:07.282731 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.282701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp" event={"ID":"31a7f6cb-d4c0-4cbc-adeb-ea37c920adae","Type":"ContainerStarted","Data":"0cf8cdaac718de8cdf0d467336c5762950c9dcd2c5dedbbe13481533377ba991"}
Jul 01 20:48:07.285992 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.285971    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd"
Jul 01 20:48:07.297165 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.297126    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-9fbqd" podStartSLOduration=2.630024351 podStartE2EDuration="4.297113755s" totalImagesPullingTime="1.667089404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:07.296398173 +0000 UTC m=+120.979882076" watchObservedRunningTime="2026-07-01 20:48:07.297113755 +0000 UTC m=+120.980597659"
Jul 01 20:48:07.331281 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:07.331235    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp" podStartSLOduration=2.070333611 podStartE2EDuration="4.33122323s" totalImagesPullingTime="2.260889619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:07.330505926 +0000 UTC m=+121.013989831" watchObservedRunningTime="2026-07-01 20:48:07.33122323 +0000 UTC m=+121.014707126"
Jul 01 20:48:16.649037 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:16.648995    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:48:16.651307 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:16.651283    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9a83a3-4849-4baf-a9a7-781bf7974983-metrics-certs\") pod \"network-metrics-daemon-7fwz5\" (UID: \"8e9a83a3-4849-4baf-a9a7-781bf7974983\") " pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:48:16.781259 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:16.781229    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xhskk\""
Jul 01 20:48:16.789050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:16.789031    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7fwz5"
Jul 01 20:48:16.916368 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:16.916296    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7fwz5"]
Jul 01 20:48:16.919592 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:48:16.919569    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9a83a3_4849_4baf_a9a7_781bf7974983.slice/crio-5e6f35c3bd4ab073eb6a98ff2ec1c9373bdcc0144f6637ad008736ab0d81ae1b WatchSource:0}: Error finding container 5e6f35c3bd4ab073eb6a98ff2ec1c9373bdcc0144f6637ad008736ab0d81ae1b: Status 404 returned error can't find the container with id 5e6f35c3bd4ab073eb6a98ff2ec1c9373bdcc0144f6637ad008736ab0d81ae1b
Jul 01 20:48:17.322749 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:17.322669    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7fwz5" event={"ID":"8e9a83a3-4849-4baf-a9a7-781bf7974983","Type":"ContainerStarted","Data":"5e6f35c3bd4ab073eb6a98ff2ec1c9373bdcc0144f6637ad008736ab0d81ae1b"}
Jul 01 20:48:18.328255 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:18.328226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7fwz5" event={"ID":"8e9a83a3-4849-4baf-a9a7-781bf7974983","Type":"ContainerStarted","Data":"39aeb760ea2104c296e23fb7995830717c111d3d9a68b9d88faa0bf7a647aa2d"}
Jul 01 20:48:18.328629 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:18.328274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7fwz5" event={"ID":"8e9a83a3-4849-4baf-a9a7-781bf7974983","Type":"ContainerStarted","Data":"36eda0b940a6097ae82402cc295c0d72c13ac03736ab7d6d9b25714b2687846c"}
Jul 01 20:48:18.348015 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:18.347974    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7fwz5" podStartSLOduration=130.390230007 podStartE2EDuration="2m11.347960264s" totalImagesPullingTime="957.730257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:48:18.345704749 +0000 UTC m=+132.029188654" watchObservedRunningTime="2026-07-01 20:48:18.347960264 +0000 UTC m=+132.031444167"
Jul 01 20:48:20.421350 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:20.421314    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:48:24.036259 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:24.036227    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:24.036597 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:24.036305    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:30.240523 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.240465    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" podUID="571220a3-6342-421d-9e8d-53949a4b87ba" containerName="registry" containerID="cri-o://6d9cb702009f8b8c7c081691e2fe70152681dcc402b23c84f4d440b47d5e994d" gracePeriod=30
Jul 01 20:48:30.376359 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.376327    2554 generic.go:356] "Generic (PLEG): container finished" podID="571220a3-6342-421d-9e8d-53949a4b87ba" containerID="6d9cb702009f8b8c7c081691e2fe70152681dcc402b23c84f4d440b47d5e994d" exitCode=0
Jul 01 20:48:30.376481 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.376374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" event={"ID":"571220a3-6342-421d-9e8d-53949a4b87ba","Type":"ContainerDied","Data":"6d9cb702009f8b8c7c081691e2fe70152681dcc402b23c84f4d440b47d5e994d"}
Jul 01 20:48:30.511009 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.510986    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:48:30.661309 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661278    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661323    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661353    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661382    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661494 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661435    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661705 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661498    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661705 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661529    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661705 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661557    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qlsv\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv\") pod \"571220a3-6342-421d-9e8d-53949a4b87ba\" (UID: \"571220a3-6342-421d-9e8d-53949a4b87ba\") "
Jul 01 20:48:30.661835 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661705    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:30.661835 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.661799    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-registry-certificates\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.662310 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.662246    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:30.663985 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.663923    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:30.664092 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.663979    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:30.664092 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.664029    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:30.664214 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.664169    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:30.664214 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.664200    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv" (OuterVolumeSpecName: "kube-api-access-4qlsv") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "kube-api-access-4qlsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:30.669674 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.669649    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "571220a3-6342-421d-9e8d-53949a4b87ba" (UID: "571220a3-6342-421d-9e8d-53949a4b87ba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:48:30.762553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762497    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-bound-sa-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762519    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/571220a3-6342-421d-9e8d-53949a4b87ba-trusted-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762528    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qlsv\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-kube-api-access-4qlsv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762539    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-image-registry-private-configuration\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762548    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/571220a3-6342-421d-9e8d-53949a4b87ba-registry-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762560    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/571220a3-6342-421d-9e8d-53949a4b87ba-installation-pull-secrets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:30.762802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:30.762572    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/571220a3-6342-421d-9e8d-53949a4b87ba-ca-trust-extracted\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:31.381561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:31.381527    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5579cbb788-phwkg" event={"ID":"571220a3-6342-421d-9e8d-53949a4b87ba","Type":"ContainerDied","Data":"863290b975659df333b161b01bf1d343748f2d4acbb7d458beebc62163fdbb89"}
Jul 01 20:48:31.381561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:31.381551    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5579cbb788-phwkg"
Jul 01 20:48:31.382050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:31.381584    2554 scope.go:117] "RemoveContainer" containerID="6d9cb702009f8b8c7c081691e2fe70152681dcc402b23c84f4d440b47d5e994d"
Jul 01 20:48:31.401557 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:31.401528    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:48:31.404757 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:31.404732    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5579cbb788-phwkg"]
Jul 01 20:48:32.772492 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:32.772461    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="571220a3-6342-421d-9e8d-53949a4b87ba" path="/var/lib/kubelet/pods/571220a3-6342-421d-9e8d-53949a4b87ba/volumes"
Jul 01 20:48:44.041001 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:44.040972    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:44.044764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:44.044742    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c78cb989c-7z9vp"
Jul 01 20:48:45.441852 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.441790    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-85b845f874-nb8dw" podUID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" containerName="console" containerID="cri-o://4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc" gracePeriod=15
Jul 01 20:48:45.688523 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.688498    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:48:45.781664 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781579    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.781664 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781624    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.781664 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781649    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsv6m\" (UniqueName: \"kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.781975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781694    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.781975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781721    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.781975 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781845    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config\") pod \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\" (UID: \"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7\") "
Jul 01 20:48:45.782129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.781969    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:45.782188 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.782135    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:45.782188 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.782142    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:45.782287 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.782199    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config" (OuterVolumeSpecName: "console-config") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:45.784075 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.784052    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m" (OuterVolumeSpecName: "kube-api-access-fsv6m") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "kube-api-access-fsv6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:45.784152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.784081    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:45.784241 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.784225    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" (UID: "726fb1f8-bd7c-4b77-8121-eaad35a3a4f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:45.882466 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.882438    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:45.882466 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.882460    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsv6m\" (UniqueName: \"kubernetes.io/projected/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-kube-api-access-fsv6m\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:45.882466 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.882469    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:45.882683 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.882478    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:45.882683 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:45.882487    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:48:46.438878 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.438843    2554 generic.go:356] "Generic (PLEG): container finished" podID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" containerID="4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc" exitCode=0
Jul 01 20:48:46.439063 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.438913    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b845f874-nb8dw"
Jul 01 20:48:46.439063 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.438929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b845f874-nb8dw" event={"ID":"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7","Type":"ContainerDied","Data":"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"}
Jul 01 20:48:46.439063 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.439035    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b845f874-nb8dw" event={"ID":"726fb1f8-bd7c-4b77-8121-eaad35a3a4f7","Type":"ContainerDied","Data":"f19ed4ad2d2a73b779333554a0c97e49766eb30461c11c36de9334603f11ecf9"}
Jul 01 20:48:46.439166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.439063    2554 scope.go:117] "RemoveContainer" containerID="4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"
Jul 01 20:48:46.447506 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.447335    2554 scope.go:117] "RemoveContainer" containerID="4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"
Jul 01 20:48:46.447712 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:48:46.447593    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc\": container with ID starting with 4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc not found: ID does not exist" containerID="4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"
Jul 01 20:48:46.447712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.447619    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc"} err="failed to get container status \"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc\": rpc error: code = NotFound desc = could not find container \"4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc\": container with ID starting with 4e3f5f7ed39ccf8d5d3123c055dd6e7ed08fef42831394256dcccee294335ccc not found: ID does not exist"
Jul 01 20:48:46.464679 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.464653    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:48:46.470035 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.470016    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b845f874-nb8dw"]
Jul 01 20:48:46.772549 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:46.772477    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" path="/var/lib/kubelet/pods/726fb1f8-bd7c-4b77-8121-eaad35a3a4f7/volumes"
Jul 01 20:48:47.444288 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:47.444257    2554 generic.go:356] "Generic (PLEG): container finished" podID="0eed4a73-7e99-4236-b9ab-9c5de5b769c6" containerID="0f870b9682265e9d108f5162b0b460e854d32daf45200545d6cbe8dfb56ee307" exitCode=0
Jul 01 20:48:47.444446 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:47.444329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq" event={"ID":"0eed4a73-7e99-4236-b9ab-9c5de5b769c6","Type":"ContainerDied","Data":"0f870b9682265e9d108f5162b0b460e854d32daf45200545d6cbe8dfb56ee307"}
Jul 01 20:48:47.444710 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:47.444694    2554 scope.go:117] "RemoveContainer" containerID="0f870b9682265e9d108f5162b0b460e854d32daf45200545d6cbe8dfb56ee307"
Jul 01 20:48:48.450463 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:48.450435    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-976gq" event={"ID":"0eed4a73-7e99-4236-b9ab-9c5de5b769c6","Type":"ContainerStarted","Data":"1355ba596a5c4ef884e4f354cae9c530488519d01c6a109d69804271c77f0db7"}
Jul 01 20:48:48.451796 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:48.451773    2554 generic.go:356] "Generic (PLEG): container finished" podID="022a57d4-a679-49c2-98ac-2c6a12178ff8" containerID="c152f41c4b7a4235cb69bc31b7fbf41031d44af2dbacab811c9855e31f400883" exitCode=0
Jul 01 20:48:48.451898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:48.451846    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv" event={"ID":"022a57d4-a679-49c2-98ac-2c6a12178ff8","Type":"ContainerDied","Data":"c152f41c4b7a4235cb69bc31b7fbf41031d44af2dbacab811c9855e31f400883"}
Jul 01 20:48:48.452113 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:48.452101    2554 scope.go:117] "RemoveContainer" containerID="c152f41c4b7a4235cb69bc31b7fbf41031d44af2dbacab811c9855e31f400883"
Jul 01 20:48:49.458279 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:48:49.458241    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-4zcjv" event={"ID":"022a57d4-a679-49c2-98ac-2c6a12178ff8","Type":"ContainerStarted","Data":"cfcb1f6d6457545ec64e010120bd19a837f098943d41acce0970f16b438e50d3"}
Jul 01 20:49:23.921019 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.920989    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571220a3-6342-421d-9e8d-53949a4b87ba" containerName="registry"
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921404    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="571220a3-6342-421d-9e8d-53949a4b87ba" containerName="registry"
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921428    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" containerName="console"
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921436    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" containerName="console"
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921506    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="726fb1f8-bd7c-4b77-8121-eaad35a3a4f7" containerName="console"
Jul 01 20:49:23.921522 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.921519    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="571220a3-6342-421d-9e8d-53949a4b87ba" containerName="registry"
Jul 01 20:49:23.926385 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.926363    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:23.929496 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.929470    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 20:49:23.929604 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.929488    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 20:49:23.929604 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.929503    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 20:49:23.929604 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.929511    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 20:49:23.930987 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.930965    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 20:49:23.931095 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.931014    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-86w5c\""
Jul 01 20:49:23.931335 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.931320    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 20:49:23.932497 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.932479    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 20:49:23.936767 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.936745    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 20:49:23.939220 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:23.939199    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:49:24.083496 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083496 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083718 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083514    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083718 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083546    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfm24\" (UniqueName: \"kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083718 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083718 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083644    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.083718 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.083664    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.184806 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.184806 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184772    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.184806 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185062 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184842    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185062 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185062 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184889    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185062 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.184956    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfm24\" (UniqueName: \"kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185601 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.185575    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.185624    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.185701    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.185830 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.185760    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.187299 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.187270    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.187412 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.187383    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.193475 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.193450    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfm24\" (UniqueName: \"kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24\") pod \"console-6f7c496454-hpg48\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") " pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.238313 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.238281    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:24.370536 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.370512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:49:24.372513 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:49:24.372477    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476b0467_88ce_4a1e_829e_93eab35aecf0.slice/crio-52845483fdbb4f74e316c74b88f1ca87a63ad939f443c28e9eff20113503f5c2 WatchSource:0}: Error finding container 52845483fdbb4f74e316c74b88f1ca87a63ad939f443c28e9eff20113503f5c2: Status 404 returned error can't find the container with id 52845483fdbb4f74e316c74b88f1ca87a63ad939f443c28e9eff20113503f5c2
Jul 01 20:49:24.587794 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.587702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7c496454-hpg48" event={"ID":"476b0467-88ce-4a1e-829e-93eab35aecf0","Type":"ContainerStarted","Data":"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"}
Jul 01 20:49:24.587794 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.587755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7c496454-hpg48" event={"ID":"476b0467-88ce-4a1e-829e-93eab35aecf0","Type":"ContainerStarted","Data":"52845483fdbb4f74e316c74b88f1ca87a63ad939f443c28e9eff20113503f5c2"}
Jul 01 20:49:24.608666 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:24.608086    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f7c496454-hpg48" podStartSLOduration=1.608066794 podStartE2EDuration="1.608066794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:49:24.60754497 +0000 UTC m=+198.291028865" watchObservedRunningTime="2026-07-01 20:49:24.608066794 +0000 UTC m=+198.291550696"
Jul 01 20:49:34.239027 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:34.238993    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:34.239394 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:34.239054    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:34.243857 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:34.243834    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:49:34.629990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:49:34.629893    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:50:12.090698 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.090657    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"]
Jul 01 20:50:12.094039 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.094022    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.096786 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.096763    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drncz\""
Jul 01 20:50:12.096924 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.096906    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 20:50:12.097904 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.097886    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 20:50:12.102729 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.102701    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"]
Jul 01 20:50:12.166737 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.166708    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.166898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.166743    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.166898 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.166778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zf9\" (UniqueName: \"kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.267748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.267715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.267748 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.267759    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.268044 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.267801    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zf9\" (UniqueName: \"kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.268213 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.268186    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.268261 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.268195    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.276148 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.276128    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zf9\" (UniqueName: \"kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.403746 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.403717    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:12.527954 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.527917    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"]
Jul 01 20:50:12.530051 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:50:12.530021    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8dc8c9_835f_4209_a7a2_450e264c1f7b.slice/crio-d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096 WatchSource:0}: Error finding container d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096: Status 404 returned error can't find the container with id d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096
Jul 01 20:50:12.772593 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:12.772519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4" event={"ID":"cc8dc8c9-835f-4209-a7a2-450e264c1f7b","Type":"ContainerStarted","Data":"d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096"}
Jul 01 20:50:19.802216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:19.802168    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerID="4b5a590b9d6dd96b82139838c7f48c83e4051ef124c582484fbb2c5c8be17d92" exitCode=0
Jul 01 20:50:19.802669 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:19.802255    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4" event={"ID":"cc8dc8c9-835f-4209-a7a2-450e264c1f7b","Type":"ContainerDied","Data":"4b5a590b9d6dd96b82139838c7f48c83e4051ef124c582484fbb2c5c8be17d92"}
Jul 01 20:50:21.812022 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:21.811986    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerID="7b94a407a4ff8ee81330f508ad3f7face7ca233a41bf78cc6a4e56fe25ea2ad3" exitCode=0
Jul 01 20:50:21.812397 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:21.812076    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4" event={"ID":"cc8dc8c9-835f-4209-a7a2-450e264c1f7b","Type":"ContainerDied","Data":"7b94a407a4ff8ee81330f508ad3f7face7ca233a41bf78cc6a4e56fe25ea2ad3"}
Jul 01 20:50:29.846419 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:29.846382    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerID="9f7c9945b4605469dbb1ccdaa8840ac18f16cd67bb973946ce960d3832b0a08d" exitCode=0
Jul 01 20:50:29.846789 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:29.846457    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4" event={"ID":"cc8dc8c9-835f-4209-a7a2-450e264c1f7b","Type":"ContainerDied","Data":"9f7c9945b4605469dbb1ccdaa8840ac18f16cd67bb973946ce960d3832b0a08d"}
Jul 01 20:50:30.972711 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:30.972687    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:31.131197 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.131116    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5zf9\" (UniqueName: \"kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9\") pod \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") "
Jul 01 20:50:31.131197 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.131164    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle\") pod \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") "
Jul 01 20:50:31.131197 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.131183    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util\") pod \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\" (UID: \"cc8dc8c9-835f-4209-a7a2-450e264c1f7b\") "
Jul 01 20:50:31.131793 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.131763    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle" (OuterVolumeSpecName: "bundle") pod "cc8dc8c9-835f-4209-a7a2-450e264c1f7b" (UID: "cc8dc8c9-835f-4209-a7a2-450e264c1f7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:50:31.133331 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.133300    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9" (OuterVolumeSpecName: "kube-api-access-k5zf9") pod "cc8dc8c9-835f-4209-a7a2-450e264c1f7b" (UID: "cc8dc8c9-835f-4209-a7a2-450e264c1f7b"). InnerVolumeSpecName "kube-api-access-k5zf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:50:31.135519 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.135494    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util" (OuterVolumeSpecName: "util") pod "cc8dc8c9-835f-4209-a7a2-450e264c1f7b" (UID: "cc8dc8c9-835f-4209-a7a2-450e264c1f7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:50:31.232050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.232018    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5zf9\" (UniqueName: \"kubernetes.io/projected/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-kube-api-access-k5zf9\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:50:31.232050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.232044    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:50:31.232050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.232054    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc8dc8c9-835f-4209-a7a2-450e264c1f7b-util\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:50:31.855418 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.855388    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4"
Jul 01 20:50:31.855575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.855378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589ks5h4" event={"ID":"cc8dc8c9-835f-4209-a7a2-450e264c1f7b","Type":"ContainerDied","Data":"d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096"}
Jul 01 20:50:31.855575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:31.855526    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d575010a0257fe87905ae2102b14d51d215fad14bdfdcf1481fb1ec0305ca096"
Jul 01 20:50:35.817645 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817612    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"]
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817922    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="util"
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817953    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="util"
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817962    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="extract"
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817968    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="extract"
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817982    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="pull"
Jul 01 20:50:35.818036 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.817988    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="pull"
Jul 01 20:50:35.818228 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.818067    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8dc8c9-835f-4209-a7a2-450e264c1f7b" containerName="extract"
Jul 01 20:50:35.821485 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.821469    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:35.826497 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.826472    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 20:50:35.826769 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.826747    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tl6sw\""
Jul 01 20:50:35.826833 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.826780    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 20:50:35.827152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.827134    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 20:50:35.844039 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.844005    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"]
Jul 01 20:50:35.968868 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.968835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5hq\" (UniqueName: \"kubernetes.io/projected/d984c549-7464-497e-b08b-0b9bd38bdb7a-kube-api-access-9f5hq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:35.969078 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:35.968885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d984c549-7464-497e-b08b-0b9bd38bdb7a-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.070409 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.070335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5hq\" (UniqueName: \"kubernetes.io/projected/d984c549-7464-497e-b08b-0b9bd38bdb7a-kube-api-access-9f5hq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.070409 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.070379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d984c549-7464-497e-b08b-0b9bd38bdb7a-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.072576 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.072556    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d984c549-7464-497e-b08b-0b9bd38bdb7a-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.082797 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.082777    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5hq\" (UniqueName: \"kubernetes.io/projected/d984c549-7464-497e-b08b-0b9bd38bdb7a-kube-api-access-9f5hq\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-blv7r\" (UID: \"d984c549-7464-497e-b08b-0b9bd38bdb7a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.130984 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.130958    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:36.266240 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:50:36.266201    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd984c549_7464_497e_b08b_0b9bd38bdb7a.slice/crio-7475fd61e6008a812f6ce52b08da603234cf78c33c71dc9dfdd52ecb0c7dad69 WatchSource:0}: Error finding container 7475fd61e6008a812f6ce52b08da603234cf78c33c71dc9dfdd52ecb0c7dad69: Status 404 returned error can't find the container with id 7475fd61e6008a812f6ce52b08da603234cf78c33c71dc9dfdd52ecb0c7dad69
Jul 01 20:50:36.266367 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.266283    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"]
Jul 01 20:50:36.876720 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:36.876680    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r" event={"ID":"d984c549-7464-497e-b08b-0b9bd38bdb7a","Type":"ContainerStarted","Data":"7475fd61e6008a812f6ce52b08da603234cf78c33c71dc9dfdd52ecb0c7dad69"}
Jul 01 20:50:38.887083 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:38.887049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r" event={"ID":"d984c549-7464-497e-b08b-0b9bd38bdb7a","Type":"ContainerStarted","Data":"20ad5df06908a41bdd36e105740a6ea248107d2626c8adb878033f11abff1ced"}
Jul 01 20:50:38.887418 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:38.887190    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:50:38.925477 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:38.925428    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r" podStartSLOduration=1.397145365 podStartE2EDuration="3.925411132s" totalImagesPullingTime="2.528265767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:50:38.923183108 +0000 UTC m=+272.606667013" watchObservedRunningTime="2026-07-01 20:50:38.925411132 +0000 UTC m=+272.608895036"
Jul 01 20:50:39.795021 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.794981    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-v2x97"]
Jul 01 20:50:39.798347 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.798329    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:39.801387 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.801368    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lsp57\""
Jul 01 20:50:39.802414 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.802401    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 20:50:39.802414 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.802412    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 20:50:39.815403 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.815380    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-v2x97"]
Jul 01 20:50:39.899745 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.899717    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-cabundle0\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:39.900122 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.899760    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6msx\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-kube-api-access-r6msx\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:39.900122 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:39.899849    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.001328 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.001292    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6msx\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-kube-api-access-r6msx\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.001525 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.001394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.001599 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.001533    2554 secret.go:281] references non-existent secret key: ca.crt
Jul 01 20:50:40.001599 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.001552    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-cabundle0\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.001731 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.001558    2554 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 20:50:40.001731 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.001663    2554 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-v2x97: references non-existent secret key: ca.crt
Jul 01 20:50:40.001731 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.001729    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates podName:8d5f4bce-7e74-4c3c-807c-abd4142c38ac nodeName:}" failed. No retries permitted until 2026-07-01 20:50:40.501711083 +0000 UTC m=+274.185194983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates") pod "keda-operator-68bff59c-v2x97" (UID: "8d5f4bce-7e74-4c3c-807c-abd4142c38ac") : references non-existent secret key: ca.crt
Jul 01 20:50:40.002276 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.002255    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-cabundle0\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.014584 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.014554    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6msx\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-kube-api-access-r6msx\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.507457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:40.507422    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:40.507629 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.507533    2554 secret.go:281] references non-existent secret key: ca.crt
Jul 01 20:50:40.507629 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.507547    2554 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 20:50:40.507629 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.507554    2554 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-v2x97: references non-existent secret key: ca.crt
Jul 01 20:50:40.507629 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:50:40.507601    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates podName:8d5f4bce-7e74-4c3c-807c-abd4142c38ac nodeName:}" failed. No retries permitted until 2026-07-01 20:50:41.507588043 +0000 UTC m=+275.191071925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates") pod "keda-operator-68bff59c-v2x97" (UID: "8d5f4bce-7e74-4c3c-807c-abd4142c38ac") : references non-existent secret key: ca.crt
Jul 01 20:50:41.517001 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:41.516959    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:41.519427 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:41.519404    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d5f4bce-7e74-4c3c-807c-abd4142c38ac-certificates\") pod \"keda-operator-68bff59c-v2x97\" (UID: \"8d5f4bce-7e74-4c3c-807c-abd4142c38ac\") " pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:41.609085 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:41.609044    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:41.743707 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:41.743677    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-v2x97"]
Jul 01 20:50:41.745670 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:50:41.745644    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5f4bce_7e74_4c3c_807c_abd4142c38ac.slice/crio-6471ba3638e18c6ec9ae2f6be26f1fceef9dc0ad4fa5849a219d765af13cf146 WatchSource:0}: Error finding container 6471ba3638e18c6ec9ae2f6be26f1fceef9dc0ad4fa5849a219d765af13cf146: Status 404 returned error can't find the container with id 6471ba3638e18c6ec9ae2f6be26f1fceef9dc0ad4fa5849a219d765af13cf146
Jul 01 20:50:41.899528 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:41.899490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-v2x97" event={"ID":"8d5f4bce-7e74-4c3c-807c-abd4142c38ac","Type":"ContainerStarted","Data":"6471ba3638e18c6ec9ae2f6be26f1fceef9dc0ad4fa5849a219d765af13cf146"}
Jul 01 20:50:45.919575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:45.919539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-v2x97" event={"ID":"8d5f4bce-7e74-4c3c-807c-abd4142c38ac","Type":"ContainerStarted","Data":"d638eca4110d3e880a95f4250151e1ede3821389af53a140270c27c2858d5f33"}
Jul 01 20:50:45.919923 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:45.919734    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:50:45.947177 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:45.947105    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-v2x97" podStartSLOduration=3.571057922 podStartE2EDuration="6.94708553s" totalImagesPullingTime="3.376027608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:50:45.946381522 +0000 UTC m=+279.629865427" watchObservedRunningTime="2026-07-01 20:50:45.94708553 +0000 UTC m=+279.630569434"
Jul 01 20:50:59.893196 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:50:59.893164    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-blv7r"
Jul 01 20:51:06.704829 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.704797    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:51:06.707325 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.707298    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:51:06.708422 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.708403    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:51:06.710553 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.710528    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:51:06.716875 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.716857    2554 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 20:51:06.926602 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:06.926577    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-v2x97"
Jul 01 20:51:45.553977 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.553877    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:51:45.557545 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.557523    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:45.560866 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.560846    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 20:51:45.560989 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.560874    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 20:51:45.561265 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.561250    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 20:51:45.562090 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.562064    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-dth2c\""
Jul 01 20:51:45.568180 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.568160    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:51:45.631024 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.630994    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:45.631151 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.631028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kgw\" (UniqueName: \"kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:45.731883 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.731852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:45.731883 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.731888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kgw\" (UniqueName: \"kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:45.732099 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:51:45.732001    2554 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 01 20:51:45.732099 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:51:45.732062    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert podName:978af4a7-3da0-48ba-8657-39b5623e2ba5 nodeName:}" failed. No retries permitted until 2026-07-01 20:51:46.232045663 +0000 UTC m=+339.915529545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert") pod "kserve-controller-manager-67485586b-rnnx4" (UID: "978af4a7-3da0-48ba-8657-39b5623e2ba5") : secret "kserve-webhook-server-cert" not found
Jul 01 20:51:45.743078 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:45.743050    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kgw\" (UniqueName: \"kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:46.236073 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:46.236039    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:46.238393 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:46.238373    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") pod \"kserve-controller-manager-67485586b-rnnx4\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") " pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:46.468440 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:46.468406    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:46.602633 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:46.602586    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:51:46.609840 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:51:46.609814    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978af4a7_3da0_48ba_8657_39b5623e2ba5.slice/crio-a392fa6e70c1bc2bb45933e512925a0e0f23280790bd94dacb17b82ebce4ffff WatchSource:0}: Error finding container a392fa6e70c1bc2bb45933e512925a0e0f23280790bd94dacb17b82ebce4ffff: Status 404 returned error can't find the container with id a392fa6e70c1bc2bb45933e512925a0e0f23280790bd94dacb17b82ebce4ffff
Jul 01 20:51:46.611095 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:46.611078    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:51:47.191650 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:47.191612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-rnnx4" event={"ID":"978af4a7-3da0-48ba-8657-39b5623e2ba5","Type":"ContainerStarted","Data":"a392fa6e70c1bc2bb45933e512925a0e0f23280790bd94dacb17b82ebce4ffff"}
Jul 01 20:51:50.207091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:50.207055    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-rnnx4" event={"ID":"978af4a7-3da0-48ba-8657-39b5623e2ba5","Type":"ContainerStarted","Data":"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"}
Jul 01 20:51:50.207456 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:50.207199    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:51:50.233605 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:51:50.233559    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-67485586b-rnnx4" podStartSLOduration=2.66973124 podStartE2EDuration="5.233544729s" totalImagesPullingTime="2.563813489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:51:50.232877882 +0000 UTC m=+343.916361787" watchObservedRunningTime="2026-07-01 20:51:50.233544729 +0000 UTC m=+343.917028637"
Jul 01 20:52:20.427547 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.427514    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:52:20.428104 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.427699    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-67485586b-rnnx4" podUID="978af4a7-3da0-48ba-8657-39b5623e2ba5" containerName="manager" containerID="cri-o://456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0" gracePeriod=10
Jul 01 20:52:20.432703 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.432680    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:52:20.451439 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.451418    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-67485586b-vgdj8"]
Jul 01 20:52:20.454540 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.454526    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.463462 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.463436    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-67485586b-vgdj8"]
Jul 01 20:52:20.608402 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.608375    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96mx\" (UniqueName: \"kubernetes.io/projected/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-kube-api-access-v96mx\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.608528 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.608426    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-cert\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.673767 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.673744    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:52:20.709166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.709096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96mx\" (UniqueName: \"kubernetes.io/projected/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-kube-api-access-v96mx\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.709166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.709155    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-cert\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.711517 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.711494    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-cert\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.717080 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.717052    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96mx\" (UniqueName: \"kubernetes.io/projected/ce354ab3-79ac-4e24-9278-7a093e9ae9cc-kube-api-access-v96mx\") pod \"kserve-controller-manager-67485586b-vgdj8\" (UID: \"ce354ab3-79ac-4e24-9278-7a093e9ae9cc\") " pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.805088 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.805046    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:20.810006 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.809983    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2kgw\" (UniqueName: \"kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw\") pod \"978af4a7-3da0-48ba-8657-39b5623e2ba5\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") "
Jul 01 20:52:20.810056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.810038    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") pod \"978af4a7-3da0-48ba-8657-39b5623e2ba5\" (UID: \"978af4a7-3da0-48ba-8657-39b5623e2ba5\") "
Jul 01 20:52:20.811979 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.811952    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert" (OuterVolumeSpecName: "cert") pod "978af4a7-3da0-48ba-8657-39b5623e2ba5" (UID: "978af4a7-3da0-48ba-8657-39b5623e2ba5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:52:20.812064 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.811975    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw" (OuterVolumeSpecName: "kube-api-access-f2kgw") pod "978af4a7-3da0-48ba-8657-39b5623e2ba5" (UID: "978af4a7-3da0-48ba-8657-39b5623e2ba5"). InnerVolumeSpecName "kube-api-access-f2kgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:52:20.911467 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.911438    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2kgw\" (UniqueName: \"kubernetes.io/projected/978af4a7-3da0-48ba-8657-39b5623e2ba5-kube-api-access-f2kgw\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:52:20.911467 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.911467    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/978af4a7-3da0-48ba-8657-39b5623e2ba5-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:52:20.927954 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:20.927916    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-67485586b-vgdj8"]
Jul 01 20:52:20.930252 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:52:20.930227    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce354ab3_79ac_4e24_9278_7a093e9ae9cc.slice/crio-326bbe3b5ff0ce2e2c9cddf167ac5e860979ddc770e014d6474aa934dfc299ff WatchSource:0}: Error finding container 326bbe3b5ff0ce2e2c9cddf167ac5e860979ddc770e014d6474aa934dfc299ff: Status 404 returned error can't find the container with id 326bbe3b5ff0ce2e2c9cddf167ac5e860979ddc770e014d6474aa934dfc299ff
Jul 01 20:52:21.339431 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.339395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-vgdj8" event={"ID":"ce354ab3-79ac-4e24-9278-7a093e9ae9cc","Type":"ContainerStarted","Data":"326bbe3b5ff0ce2e2c9cddf167ac5e860979ddc770e014d6474aa934dfc299ff"}
Jul 01 20:52:21.340586 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.340563    2554 generic.go:356] "Generic (PLEG): container finished" podID="978af4a7-3da0-48ba-8657-39b5623e2ba5" containerID="456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0" exitCode=0
Jul 01 20:52:21.340699 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.340605    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-67485586b-rnnx4"
Jul 01 20:52:21.340699 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.340646    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-rnnx4" event={"ID":"978af4a7-3da0-48ba-8657-39b5623e2ba5","Type":"ContainerDied","Data":"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"}
Jul 01 20:52:21.340699 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.340694    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-rnnx4" event={"ID":"978af4a7-3da0-48ba-8657-39b5623e2ba5","Type":"ContainerDied","Data":"a392fa6e70c1bc2bb45933e512925a0e0f23280790bd94dacb17b82ebce4ffff"}
Jul 01 20:52:21.340798 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.340717    2554 scope.go:117] "RemoveContainer" containerID="456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"
Jul 01 20:52:21.348970 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.348929    2554 scope.go:117] "RemoveContainer" containerID="456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"
Jul 01 20:52:21.349253 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:52:21.349236    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0\": container with ID starting with 456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0 not found: ID does not exist" containerID="456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"
Jul 01 20:52:21.349298 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.349264    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0"} err="failed to get container status \"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0\": rpc error: code = NotFound desc = could not find container \"456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0\": container with ID starting with 456a83812f78a9f0225daa5bfda4d81901de3ac924c1f17b4402e407531579b0 not found: ID does not exist"
Jul 01 20:52:21.361167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.361146    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:52:21.365161 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:21.365142    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-67485586b-rnnx4"]
Jul 01 20:52:22.348611 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:22.348575    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-67485586b-vgdj8" event={"ID":"ce354ab3-79ac-4e24-9278-7a093e9ae9cc","Type":"ContainerStarted","Data":"03a043225e8e0ef6304e01645771babeba5e3fead46839c1569ae0402a457301"}
Jul 01 20:52:22.349086 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:22.348665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:22.366809 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:22.366767    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-67485586b-vgdj8" podStartSLOduration=1.677189235 podStartE2EDuration="2.366753735s" totalImagesPullingTime="689.5645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:52:22.364576108 +0000 UTC m=+376.048060012" watchObservedRunningTime="2026-07-01 20:52:22.366753735 +0000 UTC m=+376.050237661"
Jul 01 20:52:22.773389 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:22.773354    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978af4a7-3da0-48ba-8657-39b5623e2ba5" path="/var/lib/kubelet/pods/978af4a7-3da0-48ba-8657-39b5623e2ba5/volumes"
Jul 01 20:52:53.359114 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:53.359081    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-67485586b-vgdj8"
Jul 01 20:52:54.362586 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.362552    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-2fkrf"]
Jul 01 20:52:54.362993 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.362863    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978af4a7-3da0-48ba-8657-39b5623e2ba5" containerName="manager"
Jul 01 20:52:54.362993 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.362874    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="978af4a7-3da0-48ba-8657-39b5623e2ba5" containerName="manager"
Jul 01 20:52:54.362993 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.362929    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="978af4a7-3da0-48ba-8657-39b5623e2ba5" containerName="manager"
Jul 01 20:52:54.365893 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.365877    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.368327 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.368302    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 20:52:54.368501 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.368485    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-d79q2\""
Jul 01 20:52:54.376525 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.376504    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2fkrf"]
Jul 01 20:52:54.381375 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.381354    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w6lg2"]
Jul 01 20:52:54.384421 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.384407    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.387008 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.386988    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 20:52:54.387094 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.387068    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-f97gr\""
Jul 01 20:52:54.395721 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.395702    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w6lg2"]
Jul 01 20:52:54.470433 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.470406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqvw\" (UniqueName: \"kubernetes.io/projected/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-kube-api-access-bjqvw\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.470563 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.470443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2999b69c-49bd-49e6-903b-335d45ca6b5e-tls-certs\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.470563 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.470530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.470563 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.470556    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndzv9\" (UniqueName: \"kubernetes.io/projected/2999b69c-49bd-49e6-903b-335d45ca6b5e-kube-api-access-ndzv9\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.571642 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.571615    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.571762 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.571645    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndzv9\" (UniqueName: \"kubernetes.io/projected/2999b69c-49bd-49e6-903b-335d45ca6b5e-kube-api-access-ndzv9\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.571762 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.571693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqvw\" (UniqueName: \"kubernetes.io/projected/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-kube-api-access-bjqvw\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.571762 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.571708    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2999b69c-49bd-49e6-903b-335d45ca6b5e-tls-certs\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.571925 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:52:54.571776    2554 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 20:52:54.571925 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:52:54.571858    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert podName:0f5d24f6-47e0-4e36-a07a-4bdfe85dea30 nodeName:}" failed. No retries permitted until 2026-07-01 20:52:55.071837423 +0000 UTC m=+408.755321325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert") pod "odh-model-controller-84b54b7f97-w6lg2" (UID: "0f5d24f6-47e0-4e36-a07a-4bdfe85dea30") : secret "odh-model-controller-webhook-cert" not found
Jul 01 20:52:54.574030 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.574012    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2999b69c-49bd-49e6-903b-335d45ca6b5e-tls-certs\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.581496 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.581469    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqvw\" (UniqueName: \"kubernetes.io/projected/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-kube-api-access-bjqvw\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:54.581820 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.581803    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndzv9\" (UniqueName: \"kubernetes.io/projected/2999b69c-49bd-49e6-903b-335d45ca6b5e-kube-api-access-ndzv9\") pod \"model-serving-api-fd65d7d6-2fkrf\" (UID: \"2999b69c-49bd-49e6-903b-335d45ca6b5e\") " pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.676364 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.676337    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:54.805457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:54.805434    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2fkrf"]
Jul 01 20:52:54.807320 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:52:54.807287    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2999b69c_49bd_49e6_903b_335d45ca6b5e.slice/crio-dc398573d9b2403c3f62d6be9b4e2b491904ed2d5b6d4f678f5bb12c7e3f1525 WatchSource:0}: Error finding container dc398573d9b2403c3f62d6be9b4e2b491904ed2d5b6d4f678f5bb12c7e3f1525: Status 404 returned error can't find the container with id dc398573d9b2403c3f62d6be9b4e2b491904ed2d5b6d4f678f5bb12c7e3f1525
Jul 01 20:52:55.076427 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.076340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:55.078653 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.078633    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f5d24f6-47e0-4e36-a07a-4bdfe85dea30-cert\") pod \"odh-model-controller-84b54b7f97-w6lg2\" (UID: \"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30\") " pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:55.296537 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.296470    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:55.464216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.464087    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-w6lg2"]
Jul 01 20:52:55.467055 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:52:55.467020    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5d24f6_47e0_4e36_a07a_4bdfe85dea30.slice/crio-c166979ee6804274343ef89bc0f5532463ecc7404e0e787cff8273b447c82203 WatchSource:0}: Error finding container c166979ee6804274343ef89bc0f5532463ecc7404e0e787cff8273b447c82203: Status 404 returned error can't find the container with id c166979ee6804274343ef89bc0f5532463ecc7404e0e787cff8273b447c82203
Jul 01 20:52:55.488606 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.488573    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2fkrf" event={"ID":"2999b69c-49bd-49e6-903b-335d45ca6b5e","Type":"ContainerStarted","Data":"dc398573d9b2403c3f62d6be9b4e2b491904ed2d5b6d4f678f5bb12c7e3f1525"}
Jul 01 20:52:55.489889 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:55.489863    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-w6lg2" event={"ID":"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30","Type":"ContainerStarted","Data":"c166979ee6804274343ef89bc0f5532463ecc7404e0e787cff8273b447c82203"}
Jul 01 20:52:58.508148 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.508114    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2fkrf" event={"ID":"2999b69c-49bd-49e6-903b-335d45ca6b5e","Type":"ContainerStarted","Data":"1f0dbd0f2aaf393e14fd630b8f81ccdfe973309c03a1850df8b2cd592c0b68fb"}
Jul 01 20:52:58.508515 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.508309    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:52:58.509747 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.509724    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-w6lg2" event={"ID":"0f5d24f6-47e0-4e36-a07a-4bdfe85dea30","Type":"ContainerStarted","Data":"f18db9d82d92cdb2eac3a218b4594cd96e43cfb2f58b47408628a8cbc303a017"}
Jul 01 20:52:58.509840 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.509822    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:52:58.527360 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.527306    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-2fkrf" podStartSLOduration=0.904626918 podStartE2EDuration="4.527289124s" totalImagesPullingTime="3.622662206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:52:58.525614376 +0000 UTC m=+412.209098282" watchObservedRunningTime="2026-07-01 20:52:58.527289124 +0000 UTC m=+412.210773028"
Jul 01 20:52:58.541663 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:52:58.541620    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-w6lg2" podStartSLOduration=1.580023612 podStartE2EDuration="4.541603328s" totalImagesPullingTime="2.961579716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:52:58.540636799 +0000 UTC m=+412.224120719" watchObservedRunningTime="2026-07-01 20:52:58.541603328 +0000 UTC m=+412.225087234"
Jul 01 20:53:09.516532 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:09.516501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-w6lg2"
Jul 01 20:53:09.518506 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:09.518488    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-2fkrf"
Jul 01 20:53:10.315450 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.315416    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-d8lbp"]
Jul 01 20:53:10.320981 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.320962    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.323432 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.323406    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 20:53:10.323547 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.323480    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vntfm\""
Jul 01 20:53:10.326523 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.326501    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-d8lbp"]
Jul 01 20:53:10.411072 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.411038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.411225 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.411082    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrlh\" (UniqueName: \"kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.512028 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.511996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.512181 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.512037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrlh\" (UniqueName: \"kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.512404 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.512375    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.520869 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.520843    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrlh\" (UniqueName: \"kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh\") pod \"s3-init-d8lbp\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") " pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.642004 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.641974    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d8lbp"
Jul 01 20:53:10.767490 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:10.767450    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-d8lbp"]
Jul 01 20:53:10.769769 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:53:10.769742    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e8d727_d72e_48c7_803a_d4aaafa9421c.slice/crio-06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9 WatchSource:0}: Error finding container 06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9: Status 404 returned error can't find the container with id 06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9
Jul 01 20:53:11.563640 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:11.563599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerStarted","Data":"06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9"}
Jul 01 20:53:14.578868 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:14.578832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerStarted","Data":"1d4b80021bb958d4751e439b2f4c77e41f67aa781b54bc3914249c48cfe2bdc9"}
Jul 01 20:53:16.589843 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:16.589808    2554 generic.go:356] "Generic (PLEG): container finished" podID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerID="1d4b80021bb958d4751e439b2f4c77e41f67aa781b54bc3914249c48cfe2bdc9" exitCode=0
Jul 01 20:53:16.590270 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:16.589885    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerDied","Data":"1d4b80021bb958d4751e439b2f4c77e41f67aa781b54bc3914249c48cfe2bdc9"}
Jul 01 20:53:20.607837 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:20.607103    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:53:21.618155 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:21.618076    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerStarted","Data":"3a2d6ebe052c225c5e2aa3b8c16406beb9dd3d30c616301d8d9b3496fe05b4d3"}
Jul 01 20:53:21.636455 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:21.636413    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-d8lbp" podStartSLOduration=2.180097004 podStartE2EDuration="11.636400124s" totalImagesPullingTime="7.796391205s" totalInitContainerRuntime="1.659911915s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:53:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:53:21.635040127 +0000 UTC m=+435.318524056" watchObservedRunningTime="2026-07-01 20:53:21.636400124 +0000 UTC m=+435.319884028"
Jul 01 20:53:45.631999 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.631959    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6f7c496454-hpg48" podUID="476b0467-88ce-4a1e-829e-93eab35aecf0" containerName="console" containerID="cri-o://e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17" gracePeriod=15
Jul 01 20:53:45.883762 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.883701    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:53:45.911877 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.911843    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.911877 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.911878    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912133 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.911915    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912133 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.911974    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912133 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912028    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfm24\" (UniqueName: \"kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912310 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912283    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca" (OuterVolumeSpecName: "service-ca") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:53:45.912310 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912301    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912419 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912343    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle\") pod \"476b0467-88ce-4a1e-829e-93eab35aecf0\" (UID: \"476b0467-88ce-4a1e-829e-93eab35aecf0\") "
Jul 01 20:53:45.912832 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912474    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config" (OuterVolumeSpecName: "console-config") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:53:45.912832 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912764    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-console-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:45.912832 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912787    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:45.913094 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.912893    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:53:45.913151 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.913101    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:53:45.914334 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.914297    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:53:45.914558 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.914533    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24" (OuterVolumeSpecName: "kube-api-access-sfm24") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "kube-api-access-sfm24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:53:45.914649 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:45.914583    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "476b0467-88ce-4a1e-829e-93eab35aecf0" (UID: "476b0467-88ce-4a1e-829e-93eab35aecf0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:53:46.013387 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.013350    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfm24\" (UniqueName: \"kubernetes.io/projected/476b0467-88ce-4a1e-829e-93eab35aecf0-kube-api-access-sfm24\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:46.013387 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.013387    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-oauth-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:46.013583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.013402    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476b0467-88ce-4a1e-829e-93eab35aecf0-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:46.013583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.013415    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-serving-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:46.013583 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.013427    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/476b0467-88ce-4a1e-829e-93eab35aecf0-console-oauth-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:53:46.734803 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.734773    2554 generic.go:356] "Generic (PLEG): container finished" podID="476b0467-88ce-4a1e-829e-93eab35aecf0" containerID="e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17" exitCode=0
Jul 01 20:53:46.735216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.734831    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7c496454-hpg48"
Jul 01 20:53:46.735216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.734831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7c496454-hpg48" event={"ID":"476b0467-88ce-4a1e-829e-93eab35aecf0","Type":"ContainerDied","Data":"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"}
Jul 01 20:53:46.735216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.735002    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7c496454-hpg48" event={"ID":"476b0467-88ce-4a1e-829e-93eab35aecf0","Type":"ContainerDied","Data":"52845483fdbb4f74e316c74b88f1ca87a63ad939f443c28e9eff20113503f5c2"}
Jul 01 20:53:46.735216 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.735033    2554 scope.go:117] "RemoveContainer" containerID="e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"
Jul 01 20:53:46.743967 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.743919    2554 scope.go:117] "RemoveContainer" containerID="e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"
Jul 01 20:53:46.744165 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:53:46.744147    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17\": container with ID starting with e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17 not found: ID does not exist" containerID="e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"
Jul 01 20:53:46.744205 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.744175    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17"} err="failed to get container status \"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17\": rpc error: code = NotFound desc = could not find container \"e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17\": container with ID starting with e3099ea02acae344b09c1a5ecdc0f8dfc14815684efe87b7847d26759d7e1a17 not found: ID does not exist"
Jul 01 20:53:46.756174 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.756151    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:53:46.760326 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.760306    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f7c496454-hpg48"]
Jul 01 20:53:46.773398 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:53:46.773376    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="476b0467-88ce-4a1e-829e-93eab35aecf0" path="/var/lib/kubelet/pods/476b0467-88ce-4a1e-829e-93eab35aecf0/volumes"
Jul 01 20:54:26.908392 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:26.908361    2554 generic.go:356] "Generic (PLEG): container finished" podID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerID="3a2d6ebe052c225c5e2aa3b8c16406beb9dd3d30c616301d8d9b3496fe05b4d3" exitCode=0
Jul 01 20:54:26.908873 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:26.908420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerDied","Data":"3a2d6ebe052c225c5e2aa3b8c16406beb9dd3d30c616301d8d9b3496fe05b4d3"}
Jul 01 20:54:28.035088 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.035064    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d8lbp"
Jul 01 20:54:28.156299 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.156273    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjrlh\" (UniqueName: \"kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh\") pod \"52e8d727-d72e-48c7-803a-d4aaafa9421c\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") "
Jul 01 20:54:28.156457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.156342    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache\") pod \"52e8d727-d72e-48c7-803a-d4aaafa9421c\" (UID: \"52e8d727-d72e-48c7-803a-d4aaafa9421c\") "
Jul 01 20:54:28.160985 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.159002    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh" (OuterVolumeSpecName: "kube-api-access-gjrlh") pod "52e8d727-d72e-48c7-803a-d4aaafa9421c" (UID: "52e8d727-d72e-48c7-803a-d4aaafa9421c"). InnerVolumeSpecName "kube-api-access-gjrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:54:28.180436 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.180409    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache" (OuterVolumeSpecName: "model-cache") pod "52e8d727-d72e-48c7-803a-d4aaafa9421c" (UID: "52e8d727-d72e-48c7-803a-d4aaafa9421c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:54:28.257611 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.257562    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrlh\" (UniqueName: \"kubernetes.io/projected/52e8d727-d72e-48c7-803a-d4aaafa9421c-kube-api-access-gjrlh\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:28.257611 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.257606    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d727-d72e-48c7-803a-d4aaafa9421c-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:28.918330 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.918298    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d8lbp" event={"ID":"52e8d727-d72e-48c7-803a-d4aaafa9421c","Type":"ContainerDied","Data":"06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9"}
Jul 01 20:54:28.918489 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.918340    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e451587df1bd3d925c6ed965cfb62b00525a248ca4344a462c72112d4c1ef9"
Jul 01 20:54:28.918489 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:28.918312    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d8lbp"
Jul 01 20:54:29.594735 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.594706    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595046    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerName="s3-init"
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595059    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerName="s3-init"
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595069    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476b0467-88ce-4a1e-829e-93eab35aecf0" containerName="console"
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595075    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="476b0467-88ce-4a1e-829e-93eab35aecf0" containerName="console"
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595082    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerName="download-hf-model"
Jul 01 20:54:29.595126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595088    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerName="download-hf-model"
Jul 01 20:54:29.595335 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595148    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e8d727-d72e-48c7-803a-d4aaafa9421c" containerName="s3-init"
Jul 01 20:54:29.595335 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.595158    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="476b0467-88ce-4a1e-829e-93eab35aecf0" containerName="console"
Jul 01 20:54:29.598167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.598150    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.600795 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.600773    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 01 20:54:29.600926 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.600806    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vntfm\""
Jul 01 20:54:29.605409 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.605240    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:29.770079 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.770046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs2k\" (UniqueName: \"kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.770079 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.770087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.871407 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.871327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs2k\" (UniqueName: \"kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.871407 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.871369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.871842 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.871822    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.880773 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.880743    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs2k\" (UniqueName: \"kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k\") pod \"seaweedfs-tls-custom-6644bf5d6c-v5vht\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:29.908374 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:29.908342    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:30.041006 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:30.040980    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:30.042817 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:54:30.042779    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cba2d4_9753_4d2f_9228_de2bbaceff57.slice/crio-dc0d67a628733eb8ad24aad7583c6246eaaf7347b891b42481cba7261935a542 WatchSource:0}: Error finding container dc0d67a628733eb8ad24aad7583c6246eaaf7347b891b42481cba7261935a542: Status 404 returned error can't find the container with id dc0d67a628733eb8ad24aad7583c6246eaaf7347b891b42481cba7261935a542
Jul 01 20:54:30.931180 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:30.931138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" event={"ID":"e9cba2d4-9753-4d2f-9228-de2bbaceff57","Type":"ContainerStarted","Data":"dc0d67a628733eb8ad24aad7583c6246eaaf7347b891b42481cba7261935a542"}
Jul 01 20:54:33.947010 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:33.946978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" event={"ID":"e9cba2d4-9753-4d2f-9228-de2bbaceff57","Type":"ContainerStarted","Data":"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"}
Jul 01 20:54:33.963837 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:33.963792    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" podStartSLOduration=1.8982711 podStartE2EDuration="4.963777026s" totalImagesPullingTime="3.065505926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:33.962974674 +0000 UTC m=+507.646458580" watchObservedRunningTime="2026-07-01 20:54:33.963777026 +0000 UTC m=+507.647260930"
Jul 01 20:54:36.217486 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:36.217445    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:36.217881 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:36.217673    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" podUID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" containerName="seaweedfs-tls-custom" containerID="cri-o://c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411" gracePeriod=30
Jul 01 20:54:37.453251 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.453227    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:37.531637 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.531555    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data\") pod \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") "
Jul 01 20:54:37.531637 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.531599    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqs2k\" (UniqueName: \"kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k\") pod \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\" (UID: \"e9cba2d4-9753-4d2f-9228-de2bbaceff57\") "
Jul 01 20:54:37.532824 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.532795    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data" (OuterVolumeSpecName: "data") pod "e9cba2d4-9753-4d2f-9228-de2bbaceff57" (UID: "e9cba2d4-9753-4d2f-9228-de2bbaceff57"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:54:37.533552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.533525    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k" (OuterVolumeSpecName: "kube-api-access-nqs2k") pod "e9cba2d4-9753-4d2f-9228-de2bbaceff57" (UID: "e9cba2d4-9753-4d2f-9228-de2bbaceff57"). InnerVolumeSpecName "kube-api-access-nqs2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:54:37.632154 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.632119    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs2k\" (UniqueName: \"kubernetes.io/projected/e9cba2d4-9753-4d2f-9228-de2bbaceff57-kube-api-access-nqs2k\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:37.632154 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.632151    2554 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9cba2d4-9753-4d2f-9228-de2bbaceff57-data\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:37.965089 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.965054    2554 generic.go:356] "Generic (PLEG): container finished" podID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" containerID="c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411" exitCode=0
Jul 01 20:54:37.965275 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.965102    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"
Jul 01 20:54:37.965275 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.965144    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" event={"ID":"e9cba2d4-9753-4d2f-9228-de2bbaceff57","Type":"ContainerDied","Data":"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"}
Jul 01 20:54:37.965275 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.965204    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht" event={"ID":"e9cba2d4-9753-4d2f-9228-de2bbaceff57","Type":"ContainerDied","Data":"dc0d67a628733eb8ad24aad7583c6246eaaf7347b891b42481cba7261935a542"}
Jul 01 20:54:37.965275 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.965233    2554 scope.go:117] "RemoveContainer" containerID="c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"
Jul 01 20:54:37.974984 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.974965    2554 scope.go:117] "RemoveContainer" containerID="c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"
Jul 01 20:54:37.975227 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:54:37.975209    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411\": container with ID starting with c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411 not found: ID does not exist" containerID="c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"
Jul 01 20:54:37.975284 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.975235    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411"} err="failed to get container status \"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411\": rpc error: code = NotFound desc = could not find container \"c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411\": container with ID starting with c49cb65215d856d537bd8bd0b9b528b48501b4826eab278717c62523ee77f411 not found: ID does not exist"
Jul 01 20:54:37.986857 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.986835    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:37.990944 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:37.990913    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-v5vht"]
Jul 01 20:54:38.021495 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.021471    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"]
Jul 01 20:54:38.021770 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.021758    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" containerName="seaweedfs-tls-custom"
Jul 01 20:54:38.021821 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.021771    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" containerName="seaweedfs-tls-custom"
Jul 01 20:54:38.021854 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.021832    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" containerName="seaweedfs-tls-custom"
Jul 01 20:54:38.025915 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.025900    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.028689 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.028651    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vntfm\""
Jul 01 20:54:38.028793 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.028696    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 01 20:54:38.028793 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.028760    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 01 20:54:38.032129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.032105    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"]
Jul 01 20:54:38.035076 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.035056    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpjq\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-kube-api-access-jvpjq\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.035166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.035085    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/571c213d-8814-4641-bf76-6abf06432878-data\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.035166 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.035104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.135570 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.135539    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpjq\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-kube-api-access-jvpjq\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.135736 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.135591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/571c213d-8814-4641-bf76-6abf06432878-data\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.135736 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.135624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.136001 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.135978    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/571c213d-8814-4641-bf76-6abf06432878-data\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.137911 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.137887    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.144782 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.144734    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpjq\" (UniqueName: \"kubernetes.io/projected/571c213d-8814-4641-bf76-6abf06432878-kube-api-access-jvpjq\") pod \"seaweedfs-tls-custom-8b44d767d-tc2x5\" (UID: \"571c213d-8814-4641-bf76-6abf06432878\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.335981 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.335869    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"
Jul 01 20:54:38.463425 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.463375    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-tc2x5"]
Jul 01 20:54:38.465574 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:54:38.465548    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571c213d_8814_4641_bf76_6abf06432878.slice/crio-ef3b6bd3088404ef9084a957cbd7607018feba60016d8f55e65230e83e2a9ba7 WatchSource:0}: Error finding container ef3b6bd3088404ef9084a957cbd7607018feba60016d8f55e65230e83e2a9ba7: Status 404 returned error can't find the container with id ef3b6bd3088404ef9084a957cbd7607018feba60016d8f55e65230e83e2a9ba7
Jul 01 20:54:38.779869 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.779832    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cba2d4-9753-4d2f-9228-de2bbaceff57" path="/var/lib/kubelet/pods/e9cba2d4-9753-4d2f-9228-de2bbaceff57/volumes"
Jul 01 20:54:38.971443 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.971406    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5" event={"ID":"571c213d-8814-4641-bf76-6abf06432878","Type":"ContainerStarted","Data":"4efc53677dbe70127a60a5e324fb33427b6c0fdcee0e80fd96db250b813dd77f"}
Jul 01 20:54:38.971615 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.971459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5" event={"ID":"571c213d-8814-4641-bf76-6abf06432878","Type":"ContainerStarted","Data":"ef3b6bd3088404ef9084a957cbd7607018feba60016d8f55e65230e83e2a9ba7"}
Jul 01 20:54:38.989806 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:38.989724    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-tc2x5" podStartSLOduration=0.723564816 podStartE2EDuration="989.706225ms" totalImagesPullingTime="266.141409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:38.989138313 +0000 UTC m=+512.672622218" watchObservedRunningTime="2026-07-01 20:54:38.989706225 +0000 UTC m=+512.673190132"
Jul 01 20:54:39.281026 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.280955    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-fb4vr"]
Jul 01 20:54:39.284256 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.284241    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.292297 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.292272    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-fb4vr"]
Jul 01 20:54:39.344511 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.344481    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.344618 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.344511    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mxh\" (UniqueName: \"kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.445149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.445114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.445306 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.445155    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mxh\" (UniqueName: \"kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.445529 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.445503    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.455121 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.455092    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mxh\" (UniqueName: \"kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh\") pod \"s3-tls-init-custom-fb4vr\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") " pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.607565 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.607488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:39.738214 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.738188    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-fb4vr"]
Jul 01 20:54:39.740346 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:54:39.740320    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9f30b2f_0d09_4d7f_bb96_c33d02253348.slice/crio-4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92 WatchSource:0}: Error finding container 4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92: Status 404 returned error can't find the container with id 4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92
Jul 01 20:54:39.979075 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.979029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerStarted","Data":"9862d41be394cf2564feed0c6ceafa9468b548f5d26fe2e04d4345f59d7c9ac5"}
Jul 01 20:54:39.979075 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:39.979080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerStarted","Data":"4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92"}
Jul 01 20:54:41.990205 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:41.990170    2554 generic.go:356] "Generic (PLEG): container finished" podID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerID="9862d41be394cf2564feed0c6ceafa9468b548f5d26fe2e04d4345f59d7c9ac5" exitCode=0
Jul 01 20:54:41.990575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:41.990245    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerDied","Data":"9862d41be394cf2564feed0c6ceafa9468b548f5d26fe2e04d4345f59d7c9ac5"}
Jul 01 20:54:42.996871 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:42.996836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerStarted","Data":"1a7412fbaf96a7743c41e2ccbcfc9eac1d7f58f07cbac6ec72e013d7c237eb12"}
Jul 01 20:54:43.033390 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:43.033334    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-fb4vr" podStartSLOduration=2.523975064 podStartE2EDuration="4.033317688s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.509342624s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:43.031557301 +0000 UTC m=+516.715041205" watchObservedRunningTime="2026-07-01 20:54:43.033317688 +0000 UTC m=+516.716801592"
Jul 01 20:54:49.030163 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:49.030124    2554 generic.go:356] "Generic (PLEG): container finished" podID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerID="1a7412fbaf96a7743c41e2ccbcfc9eac1d7f58f07cbac6ec72e013d7c237eb12" exitCode=0
Jul 01 20:54:49.030537 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:49.030198    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerDied","Data":"1a7412fbaf96a7743c41e2ccbcfc9eac1d7f58f07cbac6ec72e013d7c237eb12"}
Jul 01 20:54:50.181325 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.181302    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:50.227039 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.227012    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26mxh\" (UniqueName: \"kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh\") pod \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") "
Jul 01 20:54:50.227188 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.227049    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache\") pod \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\" (UID: \"c9f30b2f-0d09-4d7f-bb96-c33d02253348\") "
Jul 01 20:54:50.229010 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.228978    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh" (OuterVolumeSpecName: "kube-api-access-26mxh") pod "c9f30b2f-0d09-4d7f-bb96-c33d02253348" (UID: "c9f30b2f-0d09-4d7f-bb96-c33d02253348"). InnerVolumeSpecName "kube-api-access-26mxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:54:50.248206 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.248182    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9f30b2f-0d09-4d7f-bb96-c33d02253348" (UID: "c9f30b2f-0d09-4d7f-bb96-c33d02253348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:54:50.328433 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.328362    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26mxh\" (UniqueName: \"kubernetes.io/projected/c9f30b2f-0d09-4d7f-bb96-c33d02253348-kube-api-access-26mxh\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:50.328433 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:50.328390    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f30b2f-0d09-4d7f-bb96-c33d02253348-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:54:51.039766 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.039680    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-fb4vr" event={"ID":"c9f30b2f-0d09-4d7f-bb96-c33d02253348","Type":"ContainerDied","Data":"4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92"}
Jul 01 20:54:51.039766 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.039707    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-fb4vr"
Jul 01 20:54:51.039766 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.039721    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d88598180b35b398b0363fdba03a7509c4209553eb62c3e62a2c96c65810f92"
Jul 01 20:54:51.637326 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637296    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"]
Jul 01 20:54:51.637674 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637604    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerName="s3-tls-init-custom"
Jul 01 20:54:51.637674 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637615    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerName="s3-tls-init-custom"
Jul 01 20:54:51.637674 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637626    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerName="download-hf-model"
Jul 01 20:54:51.637674 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637631    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerName="download-hf-model"
Jul 01 20:54:51.637800 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.637688    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f30b2f-0d09-4d7f-bb96-c33d02253348" containerName="s3-tls-init-custom"
Jul 01 20:54:51.642305 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.642289    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.644819 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.644797    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 01 20:54:51.644919 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.644823    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 01 20:54:51.649106 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.649083    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"]
Jul 01 20:54:51.739570 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.739536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqjn\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-kube-api-access-bvqjn\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.739725 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.739582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.739725 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.739613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9f84b1-170a-448c-8318-3c0b8597ec4d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.840919 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.840886    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqjn\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-kube-api-access-bvqjn\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.841115 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.840949    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.841115 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.840976    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9f84b1-170a-448c-8318-3c0b8597ec4d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.841115 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:54:51.841097    2554 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 01 20:54:51.841115 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:54:51.841117    2554 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj: secret "seaweedfs-tls-serving" not found
Jul 01 20:54:51.841305 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:54:51.841199    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving podName:2a9f84b1-170a-448c-8318-3c0b8597ec4d nodeName:}" failed. No retries permitted until 2026-07-01 20:54:52.341181487 +0000 UTC m=+526.024665389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-zmhxj" (UID: "2a9f84b1-170a-448c-8318-3c0b8597ec4d") : secret "seaweedfs-tls-serving" not found
Jul 01 20:54:51.841348 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.841312    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9f84b1-170a-448c-8318-3c0b8597ec4d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:51.851289 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:51.851261    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqjn\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-kube-api-access-bvqjn\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:52.345310 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:52.345277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:52.347645 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:52.347612    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/2a9f84b1-170a-448c-8318-3c0b8597ec4d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zmhxj\" (UID: \"2a9f84b1-170a-448c-8318-3c0b8597ec4d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:52.552666 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:52.552631    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"
Jul 01 20:54:52.680175 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:52.680149    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj"]
Jul 01 20:54:52.682403 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:54:52.682373    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9f84b1_170a_448c_8318_3c0b8597ec4d.slice/crio-234000f3d21b1bc261b7e6e5277e5f38a28b479fbef110ddb1fadf93305ab892 WatchSource:0}: Error finding container 234000f3d21b1bc261b7e6e5277e5f38a28b479fbef110ddb1fadf93305ab892: Status 404 returned error can't find the container with id 234000f3d21b1bc261b7e6e5277e5f38a28b479fbef110ddb1fadf93305ab892
Jul 01 20:54:53.050457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.050420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj" event={"ID":"2a9f84b1-170a-448c-8318-3c0b8597ec4d","Type":"ContainerStarted","Data":"24e81c7dc4f5f6f4cf968d83c26a84f677c989be902d710b371834ac5fd95be3"}
Jul 01 20:54:53.050610 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.050478    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj" event={"ID":"2a9f84b1-170a-448c-8318-3c0b8597ec4d","Type":"ContainerStarted","Data":"234000f3d21b1bc261b7e6e5277e5f38a28b479fbef110ddb1fadf93305ab892"}
Jul 01 20:54:53.067818 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.067770    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zmhxj" podStartSLOduration=1.818408067 podStartE2EDuration="2.06775492s" totalImagesPullingTime="249.346853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:53.06720936 +0000 UTC m=+526.750693264" watchObservedRunningTime="2026-07-01 20:54:53.06775492 +0000 UTC m=+526.751238824"
Jul 01 20:54:53.581685 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.581650    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-7ltmz"]
Jul 01 20:54:53.585926 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.585909    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.594901 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.594878    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-7ltmz"]
Jul 01 20:54:53.658144 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.658108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl95b\" (UniqueName: \"kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.658307 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.658177    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.759255 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.759221    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.759697 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.759281    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl95b\" (UniqueName: \"kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.759697 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.759653    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.768561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.768535    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl95b\" (UniqueName: \"kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b\") pod \"s3-tls-init-serving-7ltmz\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") " pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:53.907658 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:53.907629    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:54:54.036087 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:54.036061    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-7ltmz"]
Jul 01 20:54:54.037658 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:54:54.037621    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf70df1be_8543_4e6b_8c48_cfc208d3bd68.slice/crio-a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6 WatchSource:0}: Error finding container a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6: Status 404 returned error can't find the container with id a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6
Jul 01 20:54:54.056912 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:54.056888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerStarted","Data":"a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6"}
Jul 01 20:54:55.064321 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:55.064287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerStarted","Data":"415eb0d4e8aabddaa03dd09d1c118079bff029f9bd8fcb4d2480a7cc19f930d8"}
Jul 01 20:54:56.069764 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:56.069679    2554 generic.go:356] "Generic (PLEG): container finished" podID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerID="415eb0d4e8aabddaa03dd09d1c118079bff029f9bd8fcb4d2480a7cc19f930d8" exitCode=0
Jul 01 20:54:56.070164 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:56.069768    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerDied","Data":"415eb0d4e8aabddaa03dd09d1c118079bff029f9bd8fcb4d2480a7cc19f930d8"}
Jul 01 20:54:57.075375 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:57.075339    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerStarted","Data":"ec95f3f68dbfbb1a10394fd20f0730cf2124a3d217f11adead05d976bf9d9ec6"}
Jul 01 20:54:57.092382 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:54:57.092334    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-7ltmz" podStartSLOduration=2.484655532 podStartE2EDuration="4.092317713s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.607662181s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:57.090681682 +0000 UTC m=+530.774165577" watchObservedRunningTime="2026-07-01 20:54:57.092317713 +0000 UTC m=+530.775801617"
Jul 01 20:55:04.112272 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:04.112237    2554 generic.go:356] "Generic (PLEG): container finished" podID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerID="ec95f3f68dbfbb1a10394fd20f0730cf2124a3d217f11adead05d976bf9d9ec6" exitCode=0
Jul 01 20:55:04.112703 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:04.112313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerDied","Data":"ec95f3f68dbfbb1a10394fd20f0730cf2124a3d217f11adead05d976bf9d9ec6"}
Jul 01 20:55:05.270994 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.270970    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:55:05.358373 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.358348    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache\") pod \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") "
Jul 01 20:55:05.358510 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.358419    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl95b\" (UniqueName: \"kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b\") pod \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\" (UID: \"f70df1be-8543-4e6b-8c48-cfc208d3bd68\") "
Jul 01 20:55:05.360353 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.360329    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b" (OuterVolumeSpecName: "kube-api-access-gl95b") pod "f70df1be-8543-4e6b-8c48-cfc208d3bd68" (UID: "f70df1be-8543-4e6b-8c48-cfc208d3bd68"). InnerVolumeSpecName "kube-api-access-gl95b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:55:05.380184 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.380126    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache" (OuterVolumeSpecName: "model-cache") pod "f70df1be-8543-4e6b-8c48-cfc208d3bd68" (UID: "f70df1be-8543-4e6b-8c48-cfc208d3bd68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:55:05.459848 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.459815    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl95b\" (UniqueName: \"kubernetes.io/projected/f70df1be-8543-4e6b-8c48-cfc208d3bd68-kube-api-access-gl95b\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:55:05.459848 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:05.459850    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70df1be-8543-4e6b-8c48-cfc208d3bd68-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:55:06.123551 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:06.123516    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-7ltmz"
Jul 01 20:55:06.123712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:06.123559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-7ltmz" event={"ID":"f70df1be-8543-4e6b-8c48-cfc208d3bd68","Type":"ContainerDied","Data":"a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6"}
Jul 01 20:55:06.123712 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:06.123602    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78f4be81e31af16ca3acf5ac84514187ee2b99ca3de8c828bea814db50a37d6"
Jul 01 20:55:17.443869 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.443835    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:55:17.444318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.444224    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerName="download-hf-model"
Jul 01 20:55:17.444318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.444237    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerName="download-hf-model"
Jul 01 20:55:17.444318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.444247    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerName="s3-tls-init-serving"
Jul 01 20:55:17.444318 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.444252    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerName="s3-tls-init-serving"
Jul 01 20:55:17.444453 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.444321    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70df1be-8543-4e6b-8c48-cfc208d3bd68" containerName="s3-tls-init-serving"
Jul 01 20:55:17.449056 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.449037    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.451842 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.451817    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 01 20:55:17.451982 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.451828    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 20:55:17.452053 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.451991    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 20:55:17.452126 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.452105    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 20:55:17.452198 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.452128    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 01 20:55:17.452198 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.452192    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dp9dt\""
Jul 01 20:55:17.452315 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.452246    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 20:55:17.458191 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.458172    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:55:17.558917 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.558883    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.558917 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.558925    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.559147 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.558970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhkv\" (UniqueName: \"kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.559147 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.559037    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.559147 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.559096    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660201 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660353 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660225    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660353 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660353 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhkv\" (UniqueName: \"kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660353 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660755 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660729    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.660919 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660899    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.661009 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.660969    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.662924 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.662901    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.669344 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.669323    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhkv\" (UniqueName: \"kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv\") pod \"isvc-sklearn-batcher-predictor-7595457646-gz6j8\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.760552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.760483    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:17.889234 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:17.889210    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:55:17.891919 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:55:17.891885    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664f832_ca88_4ba7_be79_4b3c97756ab2.slice/crio-c282772844c41b9171851e1283884bdba70729ba9dfbc24d7d1f89838c05d068 WatchSource:0}: Error finding container c282772844c41b9171851e1283884bdba70729ba9dfbc24d7d1f89838c05d068: Status 404 returned error can't find the container with id c282772844c41b9171851e1283884bdba70729ba9dfbc24d7d1f89838c05d068
Jul 01 20:55:18.178410 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:18.178379    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerStarted","Data":"6316d9a9e73dba708c5881c1e1a052895dca569f0fe99c11359d5cf0260a079d"}
Jul 01 20:55:18.178568 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:18.178441    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerStarted","Data":"c282772844c41b9171851e1283884bdba70729ba9dfbc24d7d1f89838c05d068"}
Jul 01 20:55:22.198486 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:22.198442    2554 generic.go:356] "Generic (PLEG): container finished" podID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerID="6316d9a9e73dba708c5881c1e1a052895dca569f0fe99c11359d5cf0260a079d" exitCode=0
Jul 01 20:55:22.198908 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:22.198504    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerDied","Data":"6316d9a9e73dba708c5881c1e1a052895dca569f0fe99c11359d5cf0260a079d"}
Jul 01 20:55:37.293986 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:37.293923    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerStarted","Data":"2e74ac5ed7cf06f299db99d38b809bee85ba6ad75c41d6f32b59e783d7d830ff"}
Jul 01 20:55:39.306485 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:39.306448    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerStarted","Data":"e5124a322eecc8debd1268a97e181591c51806a9900b27b58722e187cfb13646"}
Jul 01 20:55:43.328962 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:43.328848    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerStarted","Data":"bd184769b165c143071e1962909fe14eea9bc3cc91d18b87febe3279e4f42616"}
Jul 01 20:55:43.329307 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:43.329091    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:43.329307 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:43.329231    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:43.330546 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:43.330519    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:55:44.333954 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:44.333904    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:44.334427 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:44.333954    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:55:44.335091 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:44.335065    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:55:44.337866 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:44.337848    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:55:44.361345 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:44.361300    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podStartSLOduration=3.031952819 podStartE2EDuration="27.361288669s" totalImagesPullingTime="20.583675377s" totalInitContainerRuntime="3.745660473s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:55:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:43.35299459 +0000 UTC m=+577.036478490" watchObservedRunningTime="2026-07-01 20:55:44.361288669 +0000 UTC m=+578.044772572"
Jul 01 20:55:45.339125 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:45.339087    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:55:45.339515 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:45.339326    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:55:46.343561 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:46.343522    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:55:46.343995 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:46.343723    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:55:56.344289 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:56.344251    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:55:56.344741 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:55:56.344706    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:06.343575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.343532    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:56:06.344099 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.344032    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:06.740985 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.740958    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:56:06.741167 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.741061    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 20:56:06.743848 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.743825    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:56:06.744044 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:06.744032    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 20:56:16.343814 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:16.343775    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:56:16.367050 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:16.344244    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:26.344100 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:26.344053    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:56:26.344552 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:26.344504    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:36.344369 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:36.344278    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:56:36.344837 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:36.344812    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:46.343687 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:46.343630    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:56:46.344252 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:46.344107    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:56:56.344814 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:56.344776    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:56:56.345355 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:56:56.345084    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:02.752372 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.752340    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:57:02.752903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.752572    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" containerID="cri-o://2e74ac5ed7cf06f299db99d38b809bee85ba6ad75c41d6f32b59e783d7d830ff" gracePeriod=30
Jul 01 20:57:02.752903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.752622    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" containerID="cri-o://e5124a322eecc8debd1268a97e181591c51806a9900b27b58722e187cfb13646" gracePeriod=30
Jul 01 20:57:02.752903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.752633    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" containerID="cri-o://bd184769b165c143071e1962909fe14eea9bc3cc91d18b87febe3279e4f42616" gracePeriod=30
Jul 01 20:57:02.804638 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.804607    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:57:02.808549 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.808519    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:02.811463 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.811443    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 01 20:57:02.812022 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.812007    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 01 20:57:02.819456 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.819432    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:57:02.948653 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.948614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfzv\" (UniqueName: \"kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:02.948802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.948670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:02.948802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.948740    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:02.948802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.948761    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:02.948802 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:02.948799    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049508 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfzv\" (UniqueName: \"kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049508 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049508 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049825 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049535    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049825 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049572    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.049990 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.049971    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.050311 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.050289    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.050311 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.050302    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.051952 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.051919    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.057153 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.057129    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfzv\" (UniqueName: \"kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv\") pod \"isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.119168 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.119133    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:03.253805 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.253778    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:57:03.255344 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:57:03.255321    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe8d06b_18ae_408d_b664_6d7aac9ccecb.slice/crio-2794f2ebb3dc4ad2c531694f23cb1016c71f42b2cb45daf94d85d3b447d055fb WatchSource:0}: Error finding container 2794f2ebb3dc4ad2c531694f23cb1016c71f42b2cb45daf94d85d3b447d055fb: Status 404 returned error can't find the container with id 2794f2ebb3dc4ad2c531694f23cb1016c71f42b2cb45daf94d85d3b447d055fb
Jul 01 20:57:03.698407 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.698373    2554 generic.go:356] "Generic (PLEG): container finished" podID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerID="e5124a322eecc8debd1268a97e181591c51806a9900b27b58722e187cfb13646" exitCode=2
Jul 01 20:57:03.698590 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.698445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerDied","Data":"e5124a322eecc8debd1268a97e181591c51806a9900b27b58722e187cfb13646"}
Jul 01 20:57:03.700054 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.700034    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerStarted","Data":"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"}
Jul 01 20:57:03.700152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:03.700084    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerStarted","Data":"2794f2ebb3dc4ad2c531694f23cb1016c71f42b2cb45daf94d85d3b447d055fb"}
Jul 01 20:57:04.335180 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:04.335134    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:06.343847 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:06.343802    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:57:06.344296 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:06.344155    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:07.723810 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:07.723779    2554 generic.go:356] "Generic (PLEG): container finished" podID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerID="2e74ac5ed7cf06f299db99d38b809bee85ba6ad75c41d6f32b59e783d7d830ff" exitCode=0
Jul 01 20:57:07.724225 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:07.723862    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerDied","Data":"2e74ac5ed7cf06f299db99d38b809bee85ba6ad75c41d6f32b59e783d7d830ff"}
Jul 01 20:57:07.725301 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:07.725283    2554 generic.go:356] "Generic (PLEG): container finished" podID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerID="9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639" exitCode=0
Jul 01 20:57:07.725406 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:07.725315    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerDied","Data":"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"}
Jul 01 20:57:08.732385 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732350    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerStarted","Data":"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"}
Jul 01 20:57:08.732826 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerStarted","Data":"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"}
Jul 01 20:57:08.732826 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerStarted","Data":"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"}
Jul 01 20:57:08.732826 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732746    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:08.732826 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732808    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:08.733008 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.732843    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:08.734116 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.734087    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:08.734692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.734665    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:08.754844 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:08.754800    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podStartSLOduration=3.018180477 podStartE2EDuration="6.754786063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736605586s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:57:08.753582576 +0000 UTC m=+662.437066480" watchObservedRunningTime="2026-07-01 20:57:08.754786063 +0000 UTC m=+662.438269967"
Jul 01 20:57:09.334336 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:09.334292    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:09.737521 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:09.737474    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:09.737991 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:09.737959    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:14.334722 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:14.334674    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:14.335196 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:14.334830    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:14.741958 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:14.741908    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:57:14.742457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:14.742429    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:14.742726 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:14.742705    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:16.344195 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:16.344151    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:57:16.344578 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:16.344470    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:19.335184 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:19.335140    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:24.334796 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:24.334749    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:24.743172 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:24.743123    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:24.743615 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:24.743586    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:26.344038 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:26.343993    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 20:57:26.344574 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:26.344126    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:26.344574 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:26.344366    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:26.344574 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:26.344447    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:29.334624 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:29.334588    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 20:57:32.852692 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:32.852646    2554 generic.go:356] "Generic (PLEG): container finished" podID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerID="bd184769b165c143071e1962909fe14eea9bc3cc91d18b87febe3279e4f42616" exitCode=0
Jul 01 20:57:32.853157 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:32.852711    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerDied","Data":"bd184769b165c143071e1962909fe14eea9bc3cc91d18b87febe3279e4f42616"}
Jul 01 20:57:32.909220 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:32.909195    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:33.002790 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.002750    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert\") pod \"1664f832-ca88-4ba7-be79-4b3c97756ab2\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") "
Jul 01 20:57:33.002991 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.002826    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location\") pod \"1664f832-ca88-4ba7-be79-4b3c97756ab2\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") "
Jul 01 20:57:33.002991 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.002852    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzhkv\" (UniqueName: \"kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv\") pod \"1664f832-ca88-4ba7-be79-4b3c97756ab2\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") "
Jul 01 20:57:33.002991 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.002886    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config\") pod \"1664f832-ca88-4ba7-be79-4b3c97756ab2\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") "
Jul 01 20:57:33.002991 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.002916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls\") pod \"1664f832-ca88-4ba7-be79-4b3c97756ab2\" (UID: \"1664f832-ca88-4ba7-be79-4b3c97756ab2\") "
Jul 01 20:57:33.003219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.003173    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1664f832-ca88-4ba7-be79-4b3c97756ab2" (UID: "1664f832-ca88-4ba7-be79-4b3c97756ab2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:57:33.003278 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.003245    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1664f832-ca88-4ba7-be79-4b3c97756ab2" (UID: "1664f832-ca88-4ba7-be79-4b3c97756ab2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:33.003320 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.003253    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1664f832-ca88-4ba7-be79-4b3c97756ab2" (UID: "1664f832-ca88-4ba7-be79-4b3c97756ab2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:33.005067 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.005039    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1664f832-ca88-4ba7-be79-4b3c97756ab2" (UID: "1664f832-ca88-4ba7-be79-4b3c97756ab2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:57:33.005067 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.005045    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv" (OuterVolumeSpecName: "kube-api-access-jzhkv") pod "1664f832-ca88-4ba7-be79-4b3c97756ab2" (UID: "1664f832-ca88-4ba7-be79-4b3c97756ab2"). InnerVolumeSpecName "kube-api-access-jzhkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:57:33.104230 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.104188    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:57:33.104230 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.104228    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1664f832-ca88-4ba7-be79-4b3c97756ab2-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:57:33.104230 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.104238    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1664f832-ca88-4ba7-be79-4b3c97756ab2-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:57:33.104230 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.104246    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1664f832-ca88-4ba7-be79-4b3c97756ab2-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:57:33.104457 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.104254    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhkv\" (UniqueName: \"kubernetes.io/projected/1664f832-ca88-4ba7-be79-4b3c97756ab2-kube-api-access-jzhkv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:57:33.859889 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.859860    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"
Jul 01 20:57:33.859889 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.859878    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8" event={"ID":"1664f832-ca88-4ba7-be79-4b3c97756ab2","Type":"ContainerDied","Data":"c282772844c41b9171851e1283884bdba70729ba9dfbc24d7d1f89838c05d068"}
Jul 01 20:57:33.860357 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.859962    2554 scope.go:117] "RemoveContainer" containerID="bd184769b165c143071e1962909fe14eea9bc3cc91d18b87febe3279e4f42616"
Jul 01 20:57:33.869096 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.869076    2554 scope.go:117] "RemoveContainer" containerID="e5124a322eecc8debd1268a97e181591c51806a9900b27b58722e187cfb13646"
Jul 01 20:57:33.876483 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.876467    2554 scope.go:117] "RemoveContainer" containerID="2e74ac5ed7cf06f299db99d38b809bee85ba6ad75c41d6f32b59e783d7d830ff"
Jul 01 20:57:33.884897 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.884874    2554 scope.go:117] "RemoveContainer" containerID="6316d9a9e73dba708c5881c1e1a052895dca569f0fe99c11359d5cf0260a079d"
Jul 01 20:57:33.885083 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.885065    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:57:33.890520 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:33.890494    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8"]
Jul 01 20:57:34.743309 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:34.743266    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:34.743762 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:34.743739    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:34.772648 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:34.772622    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" path="/var/lib/kubelet/pods/1664f832-ca88-4ba7-be79-4b3c97756ab2/volumes"
Jul 01 20:57:44.742987 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:44.742924    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:44.743356 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:44.743293    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:57:54.742730 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:54.742688    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:57:54.743245 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:57:54.743222    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:04.742960 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:04.742863    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:58:04.743434 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:04.743340    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:14.742567 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:14.742525    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:58:14.743039 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:14.742952    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:21.768531 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:21.768501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:21.768977 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:21.768678    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:27.834957 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:27.834898    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:58:27.835536 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:27.835237    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" containerID="cri-o://df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0" gracePeriod=30
Jul 01 20:58:27.835536 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:27.835254    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" containerID="cri-o://5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3" gracePeriod=30
Jul 01 20:58:27.835536 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:27.835254    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" containerID="cri-o://8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598" gracePeriod=30
Jul 01 20:58:28.111766 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:28.111687    2554 generic.go:356] "Generic (PLEG): container finished" podID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerID="5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3" exitCode=2
Jul 01 20:58:28.111903 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:28.111761    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerDied","Data":"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"}
Jul 01 20:58:29.738380 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:29.738342    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:31.768046 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:31.767999    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:58:31.768424 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:31.768308    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:32.132215 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:32.132183    2554 generic.go:356] "Generic (PLEG): container finished" podID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerID="df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0" exitCode=0
Jul 01 20:58:32.132341 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:32.132217    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerDied","Data":"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"}
Jul 01 20:58:34.738125 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:34.738086    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:39.738297 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:39.738250    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:39.738664 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:39.738388    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:41.768102 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:41.768057    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:58:41.768464 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:41.768342    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:44.738149 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:44.738108    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:47.871532 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871498    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871864    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871878    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871890    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871896    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871908    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="storage-initializer"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871914    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="storage-initializer"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871920    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container"
Jul 01 20:58:47.871972 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.871925    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container"
Jul 01 20:58:47.872219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.872001    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kserve-container"
Jul 01 20:58:47.872219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.872010    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="kube-rbac-proxy"
Jul 01 20:58:47.872219 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.872018    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1664f832-ca88-4ba7-be79-4b3c97756ab2" containerName="agent"
Jul 01 20:58:47.875652 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.875629    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:47.878579 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.878560    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 01 20:58:47.878791 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.878777    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 01 20:58:47.885285 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.885258    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 20:58:47.930451 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.930422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:47.930575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.930473    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbl9p\" (UniqueName: \"kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:47.930575 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.930558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:47.930672 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.930627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:47.930672 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:47.930654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.031944 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.031907    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.031968    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbl9p\" (UniqueName: \"kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032582 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032557    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032709 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032688    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.032709 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.032703    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.034448 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.034433    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.040679 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.040656    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbl9p\" (UniqueName: \"kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p\") pod \"isvc-logger-predictor-5b7645fdfc-phwqx\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.186788 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.186751    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:48.325664 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:48.325579    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 20:58:48.328068 ip-10-0-129-137 kubenswrapper[2554]: W0701 20:58:48.328037    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8602bf3d_d8ee_4768_af89_a2bed90140e7.slice/crio-be26bf28ba61a9117829b339491f80b83ab7f6c20e4e2d1412b18e479b1c4f86 WatchSource:0}: Error finding container be26bf28ba61a9117829b339491f80b83ab7f6c20e4e2d1412b18e479b1c4f86: Status 404 returned error can't find the container with id be26bf28ba61a9117829b339491f80b83ab7f6c20e4e2d1412b18e479b1c4f86
Jul 01 20:58:49.210542 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:49.210506    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerStarted","Data":"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"}
Jul 01 20:58:49.211059 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:49.210587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerStarted","Data":"be26bf28ba61a9117829b339491f80b83ab7f6c20e4e2d1412b18e479b1c4f86"}
Jul 01 20:58:49.738272 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:49.738233    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:51.768801 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:51.768746    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
Jul 01 20:58:51.769239 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:51.768894    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:51.769239 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:51.769062    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:51.769239 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:51.769165    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:52.226025 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:52.225993    2554 generic.go:356] "Generic (PLEG): container finished" podID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerID="38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08" exitCode=0
Jul 01 20:58:52.226186 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:52.226067    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerDied","Data":"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"}
Jul 01 20:58:53.233749 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.233717    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerStarted","Data":"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"}
Jul 01 20:58:53.234129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.233780    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerStarted","Data":"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"}
Jul 01 20:58:53.234129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.233804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerStarted","Data":"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"}
Jul 01 20:58:53.234129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.234007    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:53.234129 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.234068    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:53.235468 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.235441    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:58:53.256423 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:53.256379    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podStartSLOduration=2.7467231290000003 podStartE2EDuration="6.256366677s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.509643548s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:58:53.254234628 +0000 UTC m=+766.937718535" watchObservedRunningTime="2026-07-01 20:58:53.256366677 +0000 UTC m=+766.939850632"
Jul 01 20:58:54.239152 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:54.239119    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:58:54.239597 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:54.239153    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:58:54.240333 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:54.240307    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:54.738015 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:54.737979    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 20:58:55.244499 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:55.244455    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:58:55.244894 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:55.244867    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:58:57.992090 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:57.992067    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:58.026351 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026322    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls\") pod \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") "
Jul 01 20:58:58.026493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026360    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert\") pod \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") "
Jul 01 20:58:58.026493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026380    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config\") pod \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") "
Jul 01 20:58:58.026493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026405    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location\") pod \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") "
Jul 01 20:58:58.026493 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026449    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qfzv\" (UniqueName: \"kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv\") pod \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\" (UID: \"6fe8d06b-18ae-408d-b664-6d7aac9ccecb\") "
Jul 01 20:58:58.026780 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026751    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fe8d06b-18ae-408d-b664-6d7aac9ccecb" (UID: "6fe8d06b-18ae-408d-b664-6d7aac9ccecb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:58:58.026863 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026749    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fe8d06b-18ae-408d-b664-6d7aac9ccecb" (UID: "6fe8d06b-18ae-408d-b664-6d7aac9ccecb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:58:58.026863 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.026804    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fe8d06b-18ae-408d-b664-6d7aac9ccecb" (UID: "6fe8d06b-18ae-408d-b664-6d7aac9ccecb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:58:58.028977 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.028954    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fe8d06b-18ae-408d-b664-6d7aac9ccecb" (UID: "6fe8d06b-18ae-408d-b664-6d7aac9ccecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:58:58.029059 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.028973    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv" (OuterVolumeSpecName: "kube-api-access-4qfzv") pod "6fe8d06b-18ae-408d-b664-6d7aac9ccecb" (UID: "6fe8d06b-18ae-408d-b664-6d7aac9ccecb"). InnerVolumeSpecName "kube-api-access-4qfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:58:58.127233 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.127150    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:58:58.127233 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.127177    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:58:58.127233 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.127191    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:58:58.127233 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.127203    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfzv\" (UniqueName: \"kubernetes.io/projected/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-kube-api-access-4qfzv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:58:58.127233 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.127215    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe8d06b-18ae-408d-b664-6d7aac9ccecb-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 20:58:58.260711 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.260667    2554 generic.go:356] "Generic (PLEG): container finished" podID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerID="8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598" exitCode=0
Jul 01 20:58:58.260897 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.260722    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"
Jul 01 20:58:58.260897 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.260766    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerDied","Data":"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"}
Jul 01 20:58:58.260897 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.260820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj" event={"ID":"6fe8d06b-18ae-408d-b664-6d7aac9ccecb","Type":"ContainerDied","Data":"2794f2ebb3dc4ad2c531694f23cb1016c71f42b2cb45daf94d85d3b447d055fb"}
Jul 01 20:58:58.260897 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.260856    2554 scope.go:117] "RemoveContainer" containerID="8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"
Jul 01 20:58:58.269658 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.269643    2554 scope.go:117] "RemoveContainer" containerID="5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"
Jul 01 20:58:58.280182 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.280161    2554 scope.go:117] "RemoveContainer" containerID="df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"
Jul 01 20:58:58.284761 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.284736    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:58:58.288420 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.288405    2554 scope.go:117] "RemoveContainer" containerID="9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"
Jul 01 20:58:58.289187 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.289168    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj"]
Jul 01 20:58:58.295565 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.295548    2554 scope.go:117] "RemoveContainer" containerID="8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"
Jul 01 20:58:58.295822 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:58:58.295804    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598\": container with ID starting with 8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598 not found: ID does not exist" containerID="8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"
Jul 01 20:58:58.295876 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.295829    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598"} err="failed to get container status \"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598\": rpc error: code = NotFound desc = could not find container \"8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598\": container with ID starting with 8f55fc9ec3c9d5d3743888c1b90283b1a9b34bc528e537a0da0b5aa20a403598 not found: ID does not exist"
Jul 01 20:58:58.295876 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.295846    2554 scope.go:117] "RemoveContainer" containerID="5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"
Jul 01 20:58:58.296199 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:58:58.296181    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3\": container with ID starting with 5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3 not found: ID does not exist" containerID="5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"
Jul 01 20:58:58.296254 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.296203    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3"} err="failed to get container status \"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3\": rpc error: code = NotFound desc = could not find container \"5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3\": container with ID starting with 5afc75122ce3e7f8ae8cc188d3bc79cf098c11e26f57dbf3eacfa411f6551de3 not found: ID does not exist"
Jul 01 20:58:58.296254 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.296222    2554 scope.go:117] "RemoveContainer" containerID="df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"
Jul 01 20:58:58.296460 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:58:58.296442    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0\": container with ID starting with df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0 not found: ID does not exist" containerID="df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"
Jul 01 20:58:58.296499 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.296467    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0"} err="failed to get container status \"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0\": rpc error: code = NotFound desc = could not find container \"df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0\": container with ID starting with df3eef70f24de9cf17a160d5adefc2859cfae11ac956ca4546e0cf05879052a0 not found: ID does not exist"
Jul 01 20:58:58.296499 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.296483    2554 scope.go:117] "RemoveContainer" containerID="9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"
Jul 01 20:58:58.296730 ip-10-0-129-137 kubenswrapper[2554]: E0701 20:58:58.296713    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639\": container with ID starting with 9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639 not found: ID does not exist" containerID="9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"
Jul 01 20:58:58.296765 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.296736    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639"} err="failed to get container status \"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639\": rpc error: code = NotFound desc = could not find container \"9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639\": container with ID starting with 9da24774b28c7a8ce4de294966e9b0122b834f6aa88831f5dba3cb94319cb639 not found: ID does not exist"
Jul 01 20:58:58.778739 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:58:58.778703    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" path="/var/lib/kubelet/pods/6fe8d06b-18ae-408d-b664-6d7aac9ccecb/volumes"
Jul 01 20:59:00.248488 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:00.248461    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 20:59:00.249146 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:00.249106    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:00.249473 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:00.249452    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:59:10.249066 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:10.249021    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:10.249472 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:10.249438    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:59:20.249193 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:20.249146    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:20.249584 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:20.249495    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:59:30.249568 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:30.249472    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:30.250054 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:30.249832    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:59:40.249147 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:40.249098    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:40.249604 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:40.249571    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:59:50.250003 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:50.249928    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 20:59:50.250525 ip-10-0-129-137 kubenswrapper[2554]: I0701 20:59:50.250422    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:00:00.249172 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:00.249128    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 21:00:00.249603 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:00.249564    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:00:10.250158 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:10.250119    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:10.250751 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:10.250568    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:23.143518 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143472    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143821    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143832    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143847    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="storage-initializer"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143852    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="storage-initializer"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143860    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143865    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143880    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent"
Jul 01 21:00:23.143924 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143885    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent"
Jul 01 21:00:23.144305 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143962    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="agent"
Jul 01 21:00:23.144305 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143974    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kserve-container"
Jul 01 21:00:23.144305 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.143982    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe8d06b-18ae-408d-b664-6d7aac9ccecb" containerName="kube-rbac-proxy"
Jul 01 21:00:23.147928 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.147908    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.150777 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.150756    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 01 21:00:23.150883 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.150758    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 01 21:00:23.152671 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.152645    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 21:00:23.152909 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.152883    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" containerID="cri-o://a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff" gracePeriod=30
Jul 01 21:00:23.152909 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.152871    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" containerID="cri-o://3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4" gracePeriod=30
Jul 01 21:00:23.152909 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.152884    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" containerID="cri-o://f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e" gracePeriod=30
Jul 01 21:00:23.158277 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.158254    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:00:23.245230 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.245194    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.245230 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.245229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.245441 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.245262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.245441 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.245343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.245441 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.245380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rlp\" (UniqueName: \"kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346186 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346154    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346186 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346189    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346391 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346391 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346391 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346325    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rlp\" (UniqueName: \"kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346535 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:23.346425    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 01 21:00:23.346535 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:23.346503    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls podName:ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff nodeName:}" failed. No retries permitted until 2026-07-01 21:00:23.846481523 +0000 UTC m=+857.529965416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 01 21:00:23.346741 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346721    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346923 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346905    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.346979 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.346959    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.355374 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.355345    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rlp\" (UniqueName: \"kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.663077 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.663042    2554 generic.go:356] "Generic (PLEG): container finished" podID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerID="a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff" exitCode=2
Jul 01 21:00:23.663228 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.663113    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerDied","Data":"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"}
Jul 01 21:00:23.851172 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.851138    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:23.853454 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:23.853424    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:24.061327 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:24.061248    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:00:24.195877 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:24.195854    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:00:24.198345 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:00:24.198319    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2103c9_ab55_4f8f_8e58_0aac1fbc20ff.slice/crio-560401fe337caee47e765fdb53747a8aabaa292a2a99fa72261a3732b71c2020 WatchSource:0}: Error finding container 560401fe337caee47e765fdb53747a8aabaa292a2a99fa72261a3732b71c2020: Status 404 returned error can't find the container with id 560401fe337caee47e765fdb53747a8aabaa292a2a99fa72261a3732b71c2020
Jul 01 21:00:24.671214 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:24.671179    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerStarted","Data":"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"}
Jul 01 21:00:24.671401 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:24.671243    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerStarted","Data":"560401fe337caee47e765fdb53747a8aabaa292a2a99fa72261a3732b71c2020"}
Jul 01 21:00:25.244729 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:25.244685    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:27.690165 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:27.690122    2554 generic.go:356] "Generic (PLEG): container finished" podID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerID="3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4" exitCode=0
Jul 01 21:00:27.690542 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:27.690199    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerDied","Data":"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"}
Jul 01 21:00:30.245287 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:30.245245    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:30.249583 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:30.249559    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 21:00:30.251191 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:30.251168    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:00:35.245098 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:35.245046    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:35.245529 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:35.245211    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:40.244761 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:40.244719    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:40.249062 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:40.249036    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 21:00:40.251675 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:40.251656    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:00:44.771656 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:44.771573    2554 generic.go:356] "Generic (PLEG): container finished" podID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerID="edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269" exitCode=0
Jul 01 21:00:44.772985 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:44.772957    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerDied","Data":"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"}
Jul 01 21:00:44.774289 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:44.774272    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 21:00:45.245615 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:45.245575    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:50.244726 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:50.244674    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 21:00:50.249090 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:50.249050    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 21:00:50.249259 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:50.249183    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:50.251831 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:50.251804    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 21:00:50.251956 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:50.251907    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:53.341303 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.341273    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:53.413271 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413169    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbl9p\" (UniqueName: \"kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p\") pod \"8602bf3d-d8ee-4768-af89-a2bed90140e7\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") "
Jul 01 21:00:53.413271 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413228    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config\") pod \"8602bf3d-d8ee-4768-af89-a2bed90140e7\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") "
Jul 01 21:00:53.413524 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413315    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location\") pod \"8602bf3d-d8ee-4768-af89-a2bed90140e7\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") "
Jul 01 21:00:53.413524 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413352    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert\") pod \"8602bf3d-d8ee-4768-af89-a2bed90140e7\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") "
Jul 01 21:00:53.413524 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413373    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls\") pod \"8602bf3d-d8ee-4768-af89-a2bed90140e7\" (UID: \"8602bf3d-d8ee-4768-af89-a2bed90140e7\") "
Jul 01 21:00:53.413714 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413678    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8602bf3d-d8ee-4768-af89-a2bed90140e7" (UID: "8602bf3d-d8ee-4768-af89-a2bed90140e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:53.413766 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413692    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8602bf3d-d8ee-4768-af89-a2bed90140e7" (UID: "8602bf3d-d8ee-4768-af89-a2bed90140e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:00:53.413766 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.413696    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8602bf3d-d8ee-4768-af89-a2bed90140e7" (UID: "8602bf3d-d8ee-4768-af89-a2bed90140e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:00:53.415420 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.415393    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p" (OuterVolumeSpecName: "kube-api-access-jbl9p") pod "8602bf3d-d8ee-4768-af89-a2bed90140e7" (UID: "8602bf3d-d8ee-4768-af89-a2bed90140e7"). InnerVolumeSpecName "kube-api-access-jbl9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:00:53.415525 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.415434    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8602bf3d-d8ee-4768-af89-a2bed90140e7" (UID: "8602bf3d-d8ee-4768-af89-a2bed90140e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:00:53.514055 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.514016    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8602bf3d-d8ee-4768-af89-a2bed90140e7-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:00:53.514055 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.514049    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:00:53.514055 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.514064    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8602bf3d-d8ee-4768-af89-a2bed90140e7-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:00:53.514315 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.514076    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbl9p\" (UniqueName: \"kubernetes.io/projected/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-api-access-jbl9p\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:00:53.514315 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.514088    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8602bf3d-d8ee-4768-af89-a2bed90140e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:00:53.833970 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.833859    2554 generic.go:356] "Generic (PLEG): container finished" podID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerID="f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e" exitCode=0
Jul 01 21:00:53.833970 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.833898    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerDied","Data":"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"}
Jul 01 21:00:53.833970 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.833958    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"
Jul 01 21:00:53.834210 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.834005    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx" event={"ID":"8602bf3d-d8ee-4768-af89-a2bed90140e7","Type":"ContainerDied","Data":"be26bf28ba61a9117829b339491f80b83ab7f6c20e4e2d1412b18e479b1c4f86"}
Jul 01 21:00:53.834210 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.834045    2554 scope.go:117] "RemoveContainer" containerID="f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"
Jul 01 21:00:53.844866 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.844846    2554 scope.go:117] "RemoveContainer" containerID="a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"
Jul 01 21:00:53.854239 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.854209    2554 scope.go:117] "RemoveContainer" containerID="3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"
Jul 01 21:00:53.861629 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.861571    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 21:00:53.864532 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.864513    2554 scope.go:117] "RemoveContainer" containerID="38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"
Jul 01 21:00:53.864867 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.864843    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx"]
Jul 01 21:00:53.874333 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.874308    2554 scope.go:117] "RemoveContainer" containerID="f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"
Jul 01 21:00:53.874680 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:53.874656    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e\": container with ID starting with f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e not found: ID does not exist" containerID="f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"
Jul 01 21:00:53.874759 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.874696    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e"} err="failed to get container status \"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e\": rpc error: code = NotFound desc = could not find container \"f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e\": container with ID starting with f9cd3df57e8ce19c422375afc9006518f9c0a965f57d94c5394f552422ac1d6e not found: ID does not exist"
Jul 01 21:00:53.874759 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.874724    2554 scope.go:117] "RemoveContainer" containerID="a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"
Jul 01 21:00:53.875063 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:53.875047    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff\": container with ID starting with a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff not found: ID does not exist" containerID="a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"
Jul 01 21:00:53.875143 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.875073    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff"} err="failed to get container status \"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff\": rpc error: code = NotFound desc = could not find container \"a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff\": container with ID starting with a70a993b1341676cb19a6c4f798d169aba1fbe9a1e68c91512f1bf42149d5eff not found: ID does not exist"
Jul 01 21:00:53.875143 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.875094    2554 scope.go:117] "RemoveContainer" containerID="3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"
Jul 01 21:00:53.875376 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:53.875355    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4\": container with ID starting with 3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4 not found: ID does not exist" containerID="3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"
Jul 01 21:00:53.875694 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.875385    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4"} err="failed to get container status \"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4\": rpc error: code = NotFound desc = could not find container \"3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4\": container with ID starting with 3a3d3970f1ac297f39d804fdb9c7c85f6d7a5f4180ef4b142d4a828b81a83ed4 not found: ID does not exist"
Jul 01 21:00:53.875694 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.875406    2554 scope.go:117] "RemoveContainer" containerID="38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"
Jul 01 21:00:53.875694 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:00:53.875646    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08\": container with ID starting with 38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08 not found: ID does not exist" containerID="38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"
Jul 01 21:00:53.875694 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:53.875672    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08"} err="failed to get container status \"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08\": rpc error: code = NotFound desc = could not find container \"38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08\": container with ID starting with 38cbb118675301d0a9c29d9fc5e6140f3478e93882cf3ca6eb48716b882a1e08 not found: ID does not exist"
Jul 01 21:00:54.774660 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:00:54.774622    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" path="/var/lib/kubelet/pods/8602bf3d-d8ee-4768-af89-a2bed90140e7/volumes"
Jul 01 21:01:06.778724 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:01:06.778635    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 21:01:06.779680 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:01:06.779660    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 21:01:06.782342 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:01:06.782324    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 21:01:06.783021 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:01:06.783000    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 21:03:05.529566 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:05.529529    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerStarted","Data":"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"}
Jul 01 21:03:06.537176 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:06.537141    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerStarted","Data":"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"}
Jul 01 21:03:06.537811 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:06.537289    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:06.569034 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:06.568989    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podStartSLOduration=2.820034087 podStartE2EDuration="2m43.568973636s" totalImagesPullingTime="2m20.612935037s" totalInitContainerRuntime="20.136004512s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:03:06.567499658 +0000 UTC m=+1020.250983561" watchObservedRunningTime="2026-07-01 21:03:06.568973636 +0000 UTC m=+1020.252457539"
Jul 01 21:03:07.542593 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:07.542561    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:07.543615 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:07.543582    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 21:03:08.547443 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:08.547397    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 21:03:13.552463 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:13.552433    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:13.553034 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:13.552918    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 21:03:23.553776 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:23.553747    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:33.302896 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.302863    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:03:33.303315 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.303139    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container" containerID="cri-o://9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16" gracePeriod=30
Jul 01 21:03:33.303315 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.303158    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kube-rbac-proxy" containerID="cri-o://d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0" gracePeriod=30
Jul 01 21:03:33.547631 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.547578    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 01 21:03:33.553665 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.553586    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 21:03:33.667548 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.667510    2554 generic.go:356] "Generic (PLEG): container finished" podID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerID="d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0" exitCode=2
Jul 01 21:03:33.667721 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:33.667587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerDied","Data":"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"}
Jul 01 21:03:34.182901 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.182873    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:34.369450 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369367    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert\") pod \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") "
Jul 01 21:03:34.369450 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") pod \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") "
Jul 01 21:03:34.369450 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369435    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location\") pod \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") "
Jul 01 21:03:34.370028 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369522    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config\") pod \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") "
Jul 01 21:03:34.370028 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369548    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8rlp\" (UniqueName: \"kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp\") pod \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\" (UID: \"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff\") "
Jul 01 21:03:34.370028 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369837    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:03:34.370028 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.369877    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:03:34.371595 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.371559    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:03:34.371707 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.371621    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp" (OuterVolumeSpecName: "kube-api-access-w8rlp") pod "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff"). InnerVolumeSpecName "kube-api-access-w8rlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:03:34.375844 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.375818    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" (UID: "ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:03:34.470181 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.470144    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:03:34.470181 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.470176    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:03:34.470395 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.470197    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:03:34.470395 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.470214    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:03:34.470395 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.470225    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rlp\" (UniqueName: \"kubernetes.io/projected/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff-kube-api-access-w8rlp\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:03:34.673798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.673766    2554 generic.go:356] "Generic (PLEG): container finished" podID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerID="9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16" exitCode=0
Jul 01 21:03:34.673975 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.673826    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"
Jul 01 21:03:34.673975 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.673851    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerDied","Data":"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"}
Jul 01 21:03:34.673975 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.673912    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs" event={"ID":"ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff","Type":"ContainerDied","Data":"560401fe337caee47e765fdb53747a8aabaa292a2a99fa72261a3732b71c2020"}
Jul 01 21:03:34.673975 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.673953    2554 scope.go:117] "RemoveContainer" containerID="d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"
Jul 01 21:03:34.683452 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.683435    2554 scope.go:117] "RemoveContainer" containerID="9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"
Jul 01 21:03:34.690804 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.690787    2554 scope.go:117] "RemoveContainer" containerID="edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"
Jul 01 21:03:34.696632 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.696609    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:03:34.699561 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.699495    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs"]
Jul 01 21:03:34.699628 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.699568    2554 scope.go:117] "RemoveContainer" containerID="d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"
Jul 01 21:03:34.699843 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:03:34.699824    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0\": container with ID starting with d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0 not found: ID does not exist" containerID="d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"
Jul 01 21:03:34.699891 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.699853    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0"} err="failed to get container status \"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0\": rpc error: code = NotFound desc = could not find container \"d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0\": container with ID starting with d4418370bc0e37ecf5ee58e8d09a34c98fed53a12752b6eebfda2085521f70d0 not found: ID does not exist"
Jul 01 21:03:34.699891 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.699870    2554 scope.go:117] "RemoveContainer" containerID="9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"
Jul 01 21:03:34.700122 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:03:34.700105    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16\": container with ID starting with 9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16 not found: ID does not exist" containerID="9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"
Jul 01 21:03:34.700169 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.700128    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16"} err="failed to get container status \"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16\": rpc error: code = NotFound desc = could not find container \"9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16\": container with ID starting with 9d8ce2d3942a39eedade54a5daf0e68767ab3c6bb008c980e4b8b4b517018c16 not found: ID does not exist"
Jul 01 21:03:34.700169 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.700145    2554 scope.go:117] "RemoveContainer" containerID="edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"
Jul 01 21:03:34.700399 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:03:34.700381    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269\": container with ID starting with edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269 not found: ID does not exist" containerID="edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"
Jul 01 21:03:34.700442 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.700404    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269"} err="failed to get container status \"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269\": rpc error: code = NotFound desc = could not find container \"edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269\": container with ID starting with edca2cba35387820dd4829a0ee1b0ff1c6e85aa0be719df4ad2c82c627b4f269 not found: ID does not exist"
Jul 01 21:03:34.772522 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:34.772491    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" path="/var/lib/kubelet/pods/ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff/volumes"
Jul 01 21:03:38.621815 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.621781    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622113    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="storage-initializer"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622125    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="storage-initializer"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622136    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622141    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622149    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622154    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622163    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622167    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622180    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622185    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622201    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="storage-initializer"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622206    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="storage-initializer"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622216    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622221    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622713 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622269    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kserve-container"
Jul 01 21:03:38.622713 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622276    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622713 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622282    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8602bf3d-d8ee-4768-af89-a2bed90140e7" containerName="agent"
Jul 01 21:03:38.622713 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622289    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kube-rbac-proxy"
Jul 01 21:03:38.622713 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.622296    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2103c9-ab55-4f8f-8e58-0aac1fbc20ff" containerName="kserve-container"
Jul 01 21:03:38.640759 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.640732    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:03:38.640898 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.640867    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.643720 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.643696    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 01 21:03:38.643831 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.643704    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 01 21:03:38.644034 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.644017    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dp9dt\""
Jul 01 21:03:38.644130 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.644033    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:03:38.644130 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.644118    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:03:38.644261 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.644245    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:03:38.644325 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.644254    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:03:38.697553 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.697517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.697718 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.697562    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.697718 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.697582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.697718 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.697614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pbm\" (UniqueName: \"kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.697718 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.697649    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798113 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798284 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798119    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798336 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pbm\" (UniqueName: \"kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798372 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798409 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798729 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798696    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.798858 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.798840    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.799066 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.799048    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.800641 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.800624    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.806666 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.806640    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pbm\" (UniqueName: \"kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm\") pod \"isvc-autogluon-v2-predictor-67b849d456-lth68\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:38.953711 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:38.953676    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:03:39.107021 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:39.106998    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:03:39.108974 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:03:39.108916    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c66012_1004_48da_84ce_16d98fb4ff7a.slice/crio-21202e1a0a42536aec0d7bd244745229bfd5a48c3d43689b297c892cfed7e80b WatchSource:0}: Error finding container 21202e1a0a42536aec0d7bd244745229bfd5a48c3d43689b297c892cfed7e80b: Status 404 returned error can't find the container with id 21202e1a0a42536aec0d7bd244745229bfd5a48c3d43689b297c892cfed7e80b
Jul 01 21:03:39.700338 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:39.700295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerStarted","Data":"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"}
Jul 01 21:03:39.700832 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:03:39.700363    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerStarted","Data":"21202e1a0a42536aec0d7bd244745229bfd5a48c3d43689b297c892cfed7e80b"}
Jul 01 21:04:08.834318 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:08.834230    2554 generic.go:356] "Generic (PLEG): container finished" podID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerID="1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68" exitCode=0
Jul 01 21:04:08.834318 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:08.834304    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerDied","Data":"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"}
Jul 01 21:04:09.840723 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:09.840684    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerStarted","Data":"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"}
Jul 01 21:04:09.841222 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:09.840758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerStarted","Data":"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"}
Jul 01 21:04:09.841222 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:09.840983    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:04:09.841222 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:09.841025    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:04:09.861880 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:09.861836    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" podStartSLOduration=2.639116123 podStartE2EDuration="31.861821764s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.222705641s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:04:09.860125923 +0000 UTC m=+1083.543609828" watchObservedRunningTime="2026-07-01 21:04:09.861821764 +0000 UTC m=+1083.545305667"
Jul 01 21:04:15.852017 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:04:15.851987    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:05:38.773584 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:38.773557    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:05:48.747100 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:48.747067    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:05:48.747498 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:48.747311    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kserve-container" containerID="cri-o://3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb" gracePeriod=30
Jul 01 21:05:48.747498 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:48.747391    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kube-rbac-proxy" containerID="cri-o://2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298" gracePeriod=30
Jul 01 21:05:49.297577 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.297540    2554 generic.go:356] "Generic (PLEG): container finished" podID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerID="2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298" exitCode=2
Jul 01 21:05:49.297778 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.297615    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerDied","Data":"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"}
Jul 01 21:05:49.590650 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.590629    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:05:49.675885 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.675853    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location\") pod \"85c66012-1004-48da-84ce-16d98fb4ff7a\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") "
Jul 01 21:05:49.676062 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.675903    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert\") pod \"85c66012-1004-48da-84ce-16d98fb4ff7a\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") "
Jul 01 21:05:49.676062 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.675921    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls\") pod \"85c66012-1004-48da-84ce-16d98fb4ff7a\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") "
Jul 01 21:05:49.676062 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.675967    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config\") pod \"85c66012-1004-48da-84ce-16d98fb4ff7a\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") "
Jul 01 21:05:49.676062 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.675989    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9pbm\" (UniqueName: \"kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm\") pod \"85c66012-1004-48da-84ce-16d98fb4ff7a\" (UID: \"85c66012-1004-48da-84ce-16d98fb4ff7a\") "
Jul 01 21:05:49.676328 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.676292    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85c66012-1004-48da-84ce-16d98fb4ff7a" (UID: "85c66012-1004-48da-84ce-16d98fb4ff7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:05:49.676409 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.676307    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85c66012-1004-48da-84ce-16d98fb4ff7a" (UID: "85c66012-1004-48da-84ce-16d98fb4ff7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:05:49.678136 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.678104    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm" (OuterVolumeSpecName: "kube-api-access-m9pbm") pod "85c66012-1004-48da-84ce-16d98fb4ff7a" (UID: "85c66012-1004-48da-84ce-16d98fb4ff7a"). InnerVolumeSpecName "kube-api-access-m9pbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:05:49.678266 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.678138    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85c66012-1004-48da-84ce-16d98fb4ff7a" (UID: "85c66012-1004-48da-84ce-16d98fb4ff7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:05:49.680345 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.680316    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85c66012-1004-48da-84ce-16d98fb4ff7a" (UID: "85c66012-1004-48da-84ce-16d98fb4ff7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:49.776851 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.776829    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:05:49.776851 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.776849    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9pbm\" (UniqueName: \"kubernetes.io/projected/85c66012-1004-48da-84ce-16d98fb4ff7a-kube-api-access-m9pbm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:05:49.777186 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.776858    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c66012-1004-48da-84ce-16d98fb4ff7a-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:05:49.777186 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.776869    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c66012-1004-48da-84ce-16d98fb4ff7a-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:05:49.777186 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:49.776877    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c66012-1004-48da-84ce-16d98fb4ff7a-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:05:50.303612 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.303573    2554 generic.go:356] "Generic (PLEG): container finished" podID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerID="3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb" exitCode=0
Jul 01 21:05:50.303807 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.303622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerDied","Data":"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"}
Jul 01 21:05:50.303807 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.303631    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"
Jul 01 21:05:50.303807 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.303682    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68" event={"ID":"85c66012-1004-48da-84ce-16d98fb4ff7a","Type":"ContainerDied","Data":"21202e1a0a42536aec0d7bd244745229bfd5a48c3d43689b297c892cfed7e80b"}
Jul 01 21:05:50.303807 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.303718    2554 scope.go:117] "RemoveContainer" containerID="2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"
Jul 01 21:05:50.314614 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.314338    2554 scope.go:117] "RemoveContainer" containerID="3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"
Jul 01 21:05:50.321796 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.321781    2554 scope.go:117] "RemoveContainer" containerID="1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"
Jul 01 21:05:50.328896 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.328877    2554 scope.go:117] "RemoveContainer" containerID="2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"
Jul 01 21:05:50.329165 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:05:50.329137    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298\": container with ID starting with 2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298 not found: ID does not exist" containerID="2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"
Jul 01 21:05:50.329236 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329172    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298"} err="failed to get container status \"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298\": rpc error: code = NotFound desc = could not find container \"2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298\": container with ID starting with 2eac426a19c7978e3ecfa69abe0e6afe82684f2b70a34b4cb8067a3c21343298 not found: ID does not exist"
Jul 01 21:05:50.329236 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329189    2554 scope.go:117] "RemoveContainer" containerID="3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"
Jul 01 21:05:50.329407 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:05:50.329389    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb\": container with ID starting with 3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb not found: ID does not exist" containerID="3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"
Jul 01 21:05:50.329466 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329410    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb"} err="failed to get container status \"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb\": rpc error: code = NotFound desc = could not find container \"3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb\": container with ID starting with 3205b552b391a2446b685cdd02e7602c51fd8d6d6d15a2869a322ef618a27cbb not found: ID does not exist"
Jul 01 21:05:50.329466 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329425    2554 scope.go:117] "RemoveContainer" containerID="1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"
Jul 01 21:05:50.329466 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329438    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:05:50.329605 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:05:50.329588    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68\": container with ID starting with 1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68 not found: ID does not exist" containerID="1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"
Jul 01 21:05:50.329642 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.329611    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68"} err="failed to get container status \"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68\": rpc error: code = NotFound desc = could not find container \"1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68\": container with ID starting with 1a688668d48758c4baf7a0779efbbf8c69f474ef99ddcbc798fb2b1da3c0ae68 not found: ID does not exist"
Jul 01 21:05:50.332732 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.332713    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68"]
Jul 01 21:05:50.772551 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:50.772518    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" path="/var/lib/kubelet/pods/85c66012-1004-48da-84ce-16d98fb4ff7a/volumes"
Jul 01 21:05:53.879451 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879418    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kserve-container"
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879753    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kserve-container"
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879764    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kube-rbac-proxy"
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879770    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kube-rbac-proxy"
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879784    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="storage-initializer"
Jul 01 21:05:53.879806 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879790    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="storage-initializer"
Jul 01 21:05:53.880020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879857    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kube-rbac-proxy"
Jul 01 21:05:53.880020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.879866    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c66012-1004-48da-84ce-16d98fb4ff7a" containerName="kserve-container"
Jul 01 21:05:53.884805 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.884788    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:53.888165 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.888138    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 01 21:05:53.888499 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.888275    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:05:53.894429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.888718    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:05:53.894429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.888872    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:05:53.894429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.889052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:05:53.894429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.889117    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 01 21:05:53.894429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.889563    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dp9dt\""
Jul 01 21:05:53.896014 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:53.895960    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:05:54.008271 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.008243    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.008436 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.008301    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.008436 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.008341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shmt\" (UniqueName: \"kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.008436 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.008372    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.008436 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.008403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.109448 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.109415    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.109601 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.109473    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.109601 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.109502    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.109601 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.109528    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shmt\" (UniqueName: \"kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.109601 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.109547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.110070 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.110043    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.110226 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.110200    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.110278 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.110263    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.111793 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.111770    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.118088 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.118064    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shmt\" (UniqueName: \"kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-6d854\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.202682 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.202656    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:05:54.331906 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:54.331882    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:05:54.333766 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:05:54.333739    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc617d8bf_5e16_4310_bd45_c444b2fe990a.slice/crio-c3ddb217559347d3aeef59085a838c9fbf82726f0b740a726fc39883a2fb2b13 WatchSource:0}: Error finding container c3ddb217559347d3aeef59085a838c9fbf82726f0b740a726fc39883a2fb2b13: Status 404 returned error can't find the container with id c3ddb217559347d3aeef59085a838c9fbf82726f0b740a726fc39883a2fb2b13
Jul 01 21:05:55.328706 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:55.328670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerStarted","Data":"7d2e9fad8a67bf94951a302176b9ff5c10c27b2995b5ae15840dd879f80858c9"}
Jul 01 21:05:55.329070 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:55.328725    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerStarted","Data":"c3ddb217559347d3aeef59085a838c9fbf82726f0b740a726fc39883a2fb2b13"}
Jul 01 21:05:59.348492 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:59.348402    2554 generic.go:356] "Generic (PLEG): container finished" podID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerID="7d2e9fad8a67bf94951a302176b9ff5c10c27b2995b5ae15840dd879f80858c9" exitCode=0
Jul 01 21:05:59.348492 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:05:59.348453    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerDied","Data":"7d2e9fad8a67bf94951a302176b9ff5c10c27b2995b5ae15840dd879f80858c9"}
Jul 01 21:06:00.356979 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:00.356925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerStarted","Data":"71662137a570587c228aa0f028bdf9c87c5e10212fdd4dcb61d9a5b4256df8a0"}
Jul 01 21:06:00.357336 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:00.357031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerStarted","Data":"b3efd67070097ce7ec5ec45dd8d0f8c97d435b2b1d6f73a0a7fc9771de3b12e7"}
Jul 01 21:06:00.357336 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:00.357229    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:06:00.380086 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:00.380037    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" podStartSLOduration=2.685018942 podStartE2EDuration="7.38002284s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.695003898s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:06:00.378255873 +0000 UTC m=+1194.061739779" watchObservedRunningTime="2026-07-01 21:06:00.38002284 +0000 UTC m=+1194.063506743"
Jul 01 21:06:01.362629 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:01.362595    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:06:06.810535 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:06.810503    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 21:06:06.813001 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:06.812978    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 21:06:06.813640 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:06.813620    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 21:06:06.816126 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:06.816108    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 21:06:07.374076 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:06:07.374051    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:07:31.771444 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:31.771409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:07:33.981220 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:33.981189    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:07:33.981681 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:33.981406    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kserve-container" containerID="cri-o://b3efd67070097ce7ec5ec45dd8d0f8c97d435b2b1d6f73a0a7fc9771de3b12e7" gracePeriod=30
Jul 01 21:07:33.981681 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:33.981452    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kube-rbac-proxy" containerID="cri-o://71662137a570587c228aa0f028bdf9c87c5e10212fdd4dcb61d9a5b4256df8a0" gracePeriod=30
Jul 01 21:07:34.791125 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:34.791093    2554 generic.go:356] "Generic (PLEG): container finished" podID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerID="71662137a570587c228aa0f028bdf9c87c5e10212fdd4dcb61d9a5b4256df8a0" exitCode=2
Jul 01 21:07:34.791125 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:34.791124    2554 generic.go:356] "Generic (PLEG): container finished" podID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerID="b3efd67070097ce7ec5ec45dd8d0f8c97d435b2b1d6f73a0a7fc9771de3b12e7" exitCode=0
Jul 01 21:07:34.791333 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:34.791150    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerDied","Data":"71662137a570587c228aa0f028bdf9c87c5e10212fdd4dcb61d9a5b4256df8a0"}
Jul 01 21:07:34.791333 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:34.791213    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerDied","Data":"b3efd67070097ce7ec5ec45dd8d0f8c97d435b2b1d6f73a0a7fc9771de3b12e7"}
Jul 01 21:07:34.835476 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:34.835451    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:07:35.009825 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.009739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location\") pod \"c617d8bf-5e16-4310-bd45-c444b2fe990a\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") "
Jul 01 21:07:35.009825 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.009774    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config\") pod \"c617d8bf-5e16-4310-bd45-c444b2fe990a\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") "
Jul 01 21:07:35.010354 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.009847    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6shmt\" (UniqueName: \"kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt\") pod \"c617d8bf-5e16-4310-bd45-c444b2fe990a\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") "
Jul 01 21:07:35.010354 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.009865    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls\") pod \"c617d8bf-5e16-4310-bd45-c444b2fe990a\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") "
Jul 01 21:07:35.010354 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.009884    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert\") pod \"c617d8bf-5e16-4310-bd45-c444b2fe990a\" (UID: \"c617d8bf-5e16-4310-bd45-c444b2fe990a\") "
Jul 01 21:07:35.010354 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.010207    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c617d8bf-5e16-4310-bd45-c444b2fe990a" (UID: "c617d8bf-5e16-4310-bd45-c444b2fe990a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:35.010512 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.010412    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c617d8bf-5e16-4310-bd45-c444b2fe990a" (UID: "c617d8bf-5e16-4310-bd45-c444b2fe990a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:35.012071 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.012045    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c617d8bf-5e16-4310-bd45-c444b2fe990a" (UID: "c617d8bf-5e16-4310-bd45-c444b2fe990a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:35.012168 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.012124    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt" (OuterVolumeSpecName: "kube-api-access-6shmt") pod "c617d8bf-5e16-4310-bd45-c444b2fe990a" (UID: "c617d8bf-5e16-4310-bd45-c444b2fe990a"). InnerVolumeSpecName "kube-api-access-6shmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:35.014213 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.014191    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c617d8bf-5e16-4310-bd45-c444b2fe990a" (UID: "c617d8bf-5e16-4310-bd45-c444b2fe990a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:07:35.111129 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.111102    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c617d8bf-5e16-4310-bd45-c444b2fe990a-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:07:35.111129 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.111127    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:07:35.111298 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.111137    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shmt\" (UniqueName: \"kubernetes.io/projected/c617d8bf-5e16-4310-bd45-c444b2fe990a-kube-api-access-6shmt\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:07:35.111298 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.111147    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c617d8bf-5e16-4310-bd45-c444b2fe990a-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:07:35.111298 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.111155    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c617d8bf-5e16-4310-bd45-c444b2fe990a-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:07:35.797493 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.797459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854" event={"ID":"c617d8bf-5e16-4310-bd45-c444b2fe990a","Type":"ContainerDied","Data":"c3ddb217559347d3aeef59085a838c9fbf82726f0b740a726fc39883a2fb2b13"}
Jul 01 21:07:35.797493 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.797487    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"
Jul 01 21:07:35.797723 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.797523    2554 scope.go:117] "RemoveContainer" containerID="71662137a570587c228aa0f028bdf9c87c5e10212fdd4dcb61d9a5b4256df8a0"
Jul 01 21:07:35.806451 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.806431    2554 scope.go:117] "RemoveContainer" containerID="b3efd67070097ce7ec5ec45dd8d0f8c97d435b2b1d6f73a0a7fc9771de3b12e7"
Jul 01 21:07:35.813677 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.813660    2554 scope.go:117] "RemoveContainer" containerID="7d2e9fad8a67bf94951a302176b9ff5c10c27b2995b5ae15840dd879f80858c9"
Jul 01 21:07:35.820783 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.820760    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:07:35.824753 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:35.824731    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854"]
Jul 01 21:07:36.773282 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:36.773250    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" path="/var/lib/kubelet/pods/c617d8bf-5e16-4310-bd45-c444b2fe990a/volumes"
Jul 01 21:07:39.112125 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112081    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112407    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kube-rbac-proxy"
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112417    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kube-rbac-proxy"
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112432    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="storage-initializer"
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112438    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="storage-initializer"
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112444    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kserve-container"
Jul 01 21:07:39.112474 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112449    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kserve-container"
Jul 01 21:07:39.112679 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112508    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kserve-container"
Jul 01 21:07:39.112679 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.112516    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c617d8bf-5e16-4310-bd45-c444b2fe990a" containerName="kube-rbac-proxy"
Jul 01 21:07:39.115822 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.115801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.118365 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.118342    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:07:39.118502 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.118390    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dp9dt\""
Jul 01 21:07:39.118502 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.118399    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:07:39.118502 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.118397    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:07:39.119374 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.119354    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 01 21:07:39.119529 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.119404    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 01 21:07:39.119529 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.119422    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:07:39.125067 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.125047    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:07:39.138539 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.138518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7cb\" (UniqueName: \"kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.138655 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.138558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.138655 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.138591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.138760 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.138676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.138760 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.138701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.239647 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.239617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7cb\" (UniqueName: \"kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.239798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.239659    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.239798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.239690    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.239798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.239723    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.239798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.239740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.240192 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.240170    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.240353 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.240334    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.240421 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.240366    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.242232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.242209    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.248000 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.247981    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7cb\" (UniqueName: \"kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.426984 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.426930    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:39.559750 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.559728    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:07:39.562210 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:07:39.562177    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d92788b_10a3_40ae_be1a_043639819cf3.slice/crio-956d50b86603e6bb2e3e268f171509de20ace252524c5722137dac30c9414f00 WatchSource:0}: Error finding container 956d50b86603e6bb2e3e268f171509de20ace252524c5722137dac30c9414f00: Status 404 returned error can't find the container with id 956d50b86603e6bb2e3e268f171509de20ace252524c5722137dac30c9414f00
Jul 01 21:07:39.817658 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.817575    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerStarted","Data":"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"}
Jul 01 21:07:39.817797 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:39.817663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerStarted","Data":"956d50b86603e6bb2e3e268f171509de20ace252524c5722137dac30c9414f00"}
Jul 01 21:07:44.842372 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:44.842334    2554 generic.go:356] "Generic (PLEG): container finished" podID="1d92788b-10a3-40ae-be1a-043639819cf3" containerID="e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2" exitCode=0
Jul 01 21:07:44.842857 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:44.842394    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerDied","Data":"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"}
Jul 01 21:07:45.852910 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:45.852876    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerStarted","Data":"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"}
Jul 01 21:07:45.853295 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:45.852955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerStarted","Data":"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"}
Jul 01 21:07:45.853295 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:45.853145    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:45.880445 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:45.880396    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" podStartSLOduration=2.347756706 podStartE2EDuration="6.880379301s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.532622595s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:45.877177461 +0000 UTC m=+1299.560661365" watchObservedRunningTime="2026-07-01 21:07:45.880379301 +0000 UTC m=+1299.563863205"
Jul 01 21:07:46.858079 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:46.858043    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:07:52.868573 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:07:52.868535    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:09:14.773077 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:14.773049    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:09:19.245157 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:19.245123    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:09:19.245580 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:19.245366    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kserve-container" containerID="cri-o://4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888" gracePeriod=30
Jul 01 21:09:19.245580 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:19.245417    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kube-rbac-proxy" containerID="cri-o://b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c" gracePeriod=30
Jul 01 21:09:19.933622 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:19.933585    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d92788b_10a3_40ae_be1a_043639819cf3.slice/crio-4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d92788b_10a3_40ae_be1a_043639819cf3.slice/crio-conmon-4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888.scope\": RecentStats: unable to find data in memory cache]"
Jul 01 21:09:20.094751 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.094715    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:09:20.227237 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227148    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q7cb\" (UniqueName: \"kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb\") pod \"1d92788b-10a3-40ae-be1a-043639819cf3\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") "
Jul 01 21:09:20.227237 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227195    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location\") pod \"1d92788b-10a3-40ae-be1a-043639819cf3\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") "
Jul 01 21:09:20.227428 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227249    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert\") pod \"1d92788b-10a3-40ae-be1a-043639819cf3\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") "
Jul 01 21:09:20.227428 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config\") pod \"1d92788b-10a3-40ae-be1a-043639819cf3\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") "
Jul 01 21:09:20.227428 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227324    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls\") pod \"1d92788b-10a3-40ae-be1a-043639819cf3\" (UID: \"1d92788b-10a3-40ae-be1a-043639819cf3\") "
Jul 01 21:09:20.227671 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227632    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d92788b-10a3-40ae-be1a-043639819cf3" (UID: "1d92788b-10a3-40ae-be1a-043639819cf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:20.227671 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.227664    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d92788b-10a3-40ae-be1a-043639819cf3" (UID: "1d92788b-10a3-40ae-be1a-043639819cf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:20.229371 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.229337    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb" (OuterVolumeSpecName: "kube-api-access-5q7cb") pod "1d92788b-10a3-40ae-be1a-043639819cf3" (UID: "1d92788b-10a3-40ae-be1a-043639819cf3"). InnerVolumeSpecName "kube-api-access-5q7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:09:20.229457 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.229380    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d92788b-10a3-40ae-be1a-043639819cf3" (UID: "1d92788b-10a3-40ae-be1a-043639819cf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:09:20.232073 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.232051    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d92788b-10a3-40ae-be1a-043639819cf3" (UID: "1d92788b-10a3-40ae-be1a-043639819cf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:09:20.286426 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286387    2554 generic.go:356] "Generic (PLEG): container finished" podID="1d92788b-10a3-40ae-be1a-043639819cf3" containerID="b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c" exitCode=2
Jul 01 21:09:20.286426 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286412    2554 generic.go:356] "Generic (PLEG): container finished" podID="1d92788b-10a3-40ae-be1a-043639819cf3" containerID="4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888" exitCode=0
Jul 01 21:09:20.287020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286445    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"
Jul 01 21:09:20.287020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286451    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerDied","Data":"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"}
Jul 01 21:09:20.287020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286517    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerDied","Data":"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"}
Jul 01 21:09:20.287020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4" event={"ID":"1d92788b-10a3-40ae-be1a-043639819cf3","Type":"ContainerDied","Data":"956d50b86603e6bb2e3e268f171509de20ace252524c5722137dac30c9414f00"}
Jul 01 21:09:20.287020 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.286587    2554 scope.go:117] "RemoveContainer" containerID="b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"
Jul 01 21:09:20.295383 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.295365    2554 scope.go:117] "RemoveContainer" containerID="4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"
Jul 01 21:09:20.302478 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.302462    2554 scope.go:117] "RemoveContainer" containerID="e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"
Jul 01 21:09:20.309093 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.309072    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:09:20.310403 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.310375    2554 scope.go:117] "RemoveContainer" containerID="b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"
Jul 01 21:09:20.310864 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:20.310691    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c\": container with ID starting with b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c not found: ID does not exist" containerID="b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"
Jul 01 21:09:20.310864 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.310732    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"} err="failed to get container status \"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c\": rpc error: code = NotFound desc = could not find container \"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c\": container with ID starting with b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c not found: ID does not exist"
Jul 01 21:09:20.310864 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.310755    2554 scope.go:117] "RemoveContainer" containerID="4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"
Jul 01 21:09:20.311162 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:20.311115    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888\": container with ID starting with 4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888 not found: ID does not exist" containerID="4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"
Jul 01 21:09:20.311272 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311170    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"} err="failed to get container status \"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888\": rpc error: code = NotFound desc = could not find container \"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888\": container with ID starting with 4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888 not found: ID does not exist"
Jul 01 21:09:20.311272 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311192    2554 scope.go:117] "RemoveContainer" containerID="e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"
Jul 01 21:09:20.311485 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:20.311465    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2\": container with ID starting with e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2 not found: ID does not exist" containerID="e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"
Jul 01 21:09:20.311554 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311492    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"} err="failed to get container status \"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2\": rpc error: code = NotFound desc = could not find container \"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2\": container with ID starting with e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2 not found: ID does not exist"
Jul 01 21:09:20.311554 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311508    2554 scope.go:117] "RemoveContainer" containerID="b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"
Jul 01 21:09:20.311748 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311733    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c"} err="failed to get container status \"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c\": rpc error: code = NotFound desc = could not find container \"b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c\": container with ID starting with b7d6c9150a68ca4287434123d612f85a8166b78a250315db9607e1ccc933ef2c not found: ID does not exist"
Jul 01 21:09:20.311792 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.311749    2554 scope.go:117] "RemoveContainer" containerID="4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"
Jul 01 21:09:20.312031 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.312005    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888"} err="failed to get container status \"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888\": rpc error: code = NotFound desc = could not find container \"4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888\": container with ID starting with 4e767a7afa5e378740ef7c992683fe991d3f81ad8bed5256a13f0bbe4b431888 not found: ID does not exist"
Jul 01 21:09:20.312085 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.312034    2554 scope.go:117] "RemoveContainer" containerID="e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"
Jul 01 21:09:20.312281 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.312262    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2"} err="failed to get container status \"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2\": rpc error: code = NotFound desc = could not find container \"e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2\": container with ID starting with e6c319a999d2277d072fb674d811ea8c440aea884097af81d73c0583aaef36b2 not found: ID does not exist"
Jul 01 21:09:20.312913 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.312899    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4"]
Jul 01 21:09:20.328700 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.328682    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:20.328773 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.328702    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d92788b-10a3-40ae-be1a-043639819cf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:20.328773 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.328713    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d92788b-10a3-40ae-be1a-043639819cf3-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:20.328773 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.328722    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7cb\" (UniqueName: \"kubernetes.io/projected/1d92788b-10a3-40ae-be1a-043639819cf3-kube-api-access-5q7cb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:20.328773 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.328731    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d92788b-10a3-40ae-be1a-043639819cf3-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:20.773205 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:20.773173    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" path="/var/lib/kubelet/pods/1d92788b-10a3-40ae-be1a-043639819cf3/volumes"
Jul 01 21:09:24.388353 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388315    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388687    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="storage-initializer"
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388698    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="storage-initializer"
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388710    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kserve-container"
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388715    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kserve-container"
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388730    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kube-rbac-proxy"
Jul 01 21:09:24.388743 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388736    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kube-rbac-proxy"
Jul 01 21:09:24.388963 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388804    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kserve-container"
Jul 01 21:09:24.388963 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.388813    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d92788b-10a3-40ae-be1a-043639819cf3" containerName="kube-rbac-proxy"
Jul 01 21:09:24.393706 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.393686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.396685 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396662    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:09:24.396926 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396893    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 01 21:09:24.396926 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396913    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:09:24.397164 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396894    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dp9dt\""
Jul 01 21:09:24.397164 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396899    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:09:24.397164 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.396894    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:09:24.397164 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.397009    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 01 21:09:24.401787 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.401764    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:24.465076 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.465045    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.465264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.465098    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.465264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.465131    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.465264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.465185    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.465264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.465211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tjs\" (UniqueName: \"kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566150 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566118    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566167    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tjs\" (UniqueName: \"kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566621 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566601    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.566905 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.566881    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.567071 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.567053    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.568758 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.568738    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.576254 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.576233    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tjs\" (UniqueName: \"kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.705797 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.705712    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:24.849765 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:24.849741    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:24.852465 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:09:24.852439    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0ae410_7edd_4422_b989_2b422e80bae8.slice/crio-1c489d02a47fc1fb1a3ad50ad11b428687c8df4d1d0ee33273cacb0a0ab1dc8c WatchSource:0}: Error finding container 1c489d02a47fc1fb1a3ad50ad11b428687c8df4d1d0ee33273cacb0a0ab1dc8c: Status 404 returned error can't find the container with id 1c489d02a47fc1fb1a3ad50ad11b428687c8df4d1d0ee33273cacb0a0ab1dc8c
Jul 01 21:09:25.312802 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:25.312759    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerStarted","Data":"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"}
Jul 01 21:09:25.313014 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:25.312821    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerStarted","Data":"1c489d02a47fc1fb1a3ad50ad11b428687c8df4d1d0ee33273cacb0a0ab1dc8c"}
Jul 01 21:09:35.358052 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:35.358011    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerID="78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301" exitCode=0
Jul 01 21:09:35.358421 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:35.358077    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerDied","Data":"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"}
Jul 01 21:09:36.365702 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:36.365670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerStarted","Data":"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"}
Jul 01 21:09:36.366094 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:36.365738    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerStarted","Data":"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"}
Jul 01 21:09:36.366094 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:36.365928    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:36.387301 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:36.387240    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podStartSLOduration=2.955127953 podStartE2EDuration="12.387220404s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.432092451s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:09:36.38429029 +0000 UTC m=+1410.067774206" watchObservedRunningTime="2026-07-01 21:09:36.387220404 +0000 UTC m=+1410.070704310"
Jul 01 21:09:37.371810 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:37.371778    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:37.372755 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:37.372723    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 21:09:38.377575 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:38.377536    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 21:09:43.382708 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:43.382673    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:43.383432 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:43.383407    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 21:09:53.384681 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:53.384652    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:54.449684 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:54.449654    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:54.450151 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:54.449890    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container" containerID="cri-o://abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1" gracePeriod=30
Jul 01 21:09:54.450151 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:54.449961    2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kube-rbac-proxy" containerID="cri-o://e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016" gracePeriod=30
Jul 01 21:09:55.289372 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.289344    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:55.326487 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326462    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location\") pod \"4d0ae410-7edd-4422-b989-2b422e80bae8\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") "
Jul 01 21:09:55.326622 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326501    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls\") pod \"4d0ae410-7edd-4422-b989-2b422e80bae8\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") "
Jul 01 21:09:55.326622 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326537    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config\") pod \"4d0ae410-7edd-4422-b989-2b422e80bae8\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") "
Jul 01 21:09:55.326622 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69tjs\" (UniqueName: \"kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs\") pod \"4d0ae410-7edd-4422-b989-2b422e80bae8\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") "
Jul 01 21:09:55.326622 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326579    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert\") pod \"4d0ae410-7edd-4422-b989-2b422e80bae8\" (UID: \"4d0ae410-7edd-4422-b989-2b422e80bae8\") "
Jul 01 21:09:55.327007 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326973    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d0ae410-7edd-4422-b989-2b422e80bae8" (UID: "4d0ae410-7edd-4422-b989-2b422e80bae8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:55.327106 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.326984    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d0ae410-7edd-4422-b989-2b422e80bae8" (UID: "4d0ae410-7edd-4422-b989-2b422e80bae8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:09:55.327485 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.327463    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d0ae410-7edd-4422-b989-2b422e80bae8" (UID: "4d0ae410-7edd-4422-b989-2b422e80bae8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:09:55.328680 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.328652    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs" (OuterVolumeSpecName: "kube-api-access-69tjs") pod "4d0ae410-7edd-4422-b989-2b422e80bae8" (UID: "4d0ae410-7edd-4422-b989-2b422e80bae8"). InnerVolumeSpecName "kube-api-access-69tjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:09:55.328788 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.328680    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d0ae410-7edd-4422-b989-2b422e80bae8" (UID: "4d0ae410-7edd-4422-b989-2b422e80bae8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:09:55.428132 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.428111    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0ae410-7edd-4422-b989-2b422e80bae8-kserve-provision-location\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:55.428132 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.428133    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0ae410-7edd-4422-b989-2b422e80bae8-proxy-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:55.428264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.428143    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:55.428264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.428152    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tjs\" (UniqueName: \"kubernetes.io/projected/4d0ae410-7edd-4422-b989-2b422e80bae8-kube-api-access-69tjs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:55.428264 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.428160    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ae410-7edd-4422-b989-2b422e80bae8-cabundle-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Jul 01 21:09:55.462394 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462367    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerID="e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016" exitCode=2
Jul 01 21:09:55.462394 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462393    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerID="abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1" exitCode=0
Jul 01 21:09:55.462731 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462428    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"
Jul 01 21:09:55.462731 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462439    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerDied","Data":"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"}
Jul 01 21:09:55.462731 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462494    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerDied","Data":"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"}
Jul 01 21:09:55.462731 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462514    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh" event={"ID":"4d0ae410-7edd-4422-b989-2b422e80bae8","Type":"ContainerDied","Data":"1c489d02a47fc1fb1a3ad50ad11b428687c8df4d1d0ee33273cacb0a0ab1dc8c"}
Jul 01 21:09:55.462731 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.462537    2554 scope.go:117] "RemoveContainer" containerID="e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"
Jul 01 21:09:55.470964 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.470945    2554 scope.go:117] "RemoveContainer" containerID="abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"
Jul 01 21:09:55.478653 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.478637    2554 scope.go:117] "RemoveContainer" containerID="78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"
Jul 01 21:09:55.487612 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.487588    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:55.490440 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.490421    2554 scope.go:117] "RemoveContainer" containerID="e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"
Jul 01 21:09:55.490849 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:55.490814    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016\": container with ID starting with e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016 not found: ID does not exist" containerID="e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"
Jul 01 21:09:55.491019 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.490988    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"} err="failed to get container status \"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016\": rpc error: code = NotFound desc = could not find container \"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016\": container with ID starting with e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016 not found: ID does not exist"
Jul 01 21:09:55.491106 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491022    2554 scope.go:117] "RemoveContainer" containerID="abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"
Jul 01 21:09:55.491359 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:55.491339    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1\": container with ID starting with abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1 not found: ID does not exist" containerID="abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"
Jul 01 21:09:55.491439 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491366    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"} err="failed to get container status \"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1\": rpc error: code = NotFound desc = could not find container \"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1\": container with ID starting with abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1 not found: ID does not exist"
Jul 01 21:09:55.491439 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491387    2554 scope.go:117] "RemoveContainer" containerID="78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"
Jul 01 21:09:55.491640 ip-10-0-129-137 kubenswrapper[2554]: E0701 21:09:55.491623    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301\": container with ID starting with 78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301 not found: ID does not exist" containerID="78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"
Jul 01 21:09:55.491681 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491646    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"} err="failed to get container status \"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301\": rpc error: code = NotFound desc = could not find container \"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301\": container with ID starting with 78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301 not found: ID does not exist"
Jul 01 21:09:55.491681 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491659    2554 scope.go:117] "RemoveContainer" containerID="e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"
Jul 01 21:09:55.491904 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491881    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016"} err="failed to get container status \"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016\": rpc error: code = NotFound desc = could not find container \"e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016\": container with ID starting with e11cecf6b8a6873827b49e791b91dea9d112293366f3e4c807db86736c6c7016 not found: ID does not exist"
Jul 01 21:09:55.491977 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.491907    2554 scope.go:117] "RemoveContainer" containerID="abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"
Jul 01 21:09:55.492187 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.492169    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1"} err="failed to get container status \"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1\": rpc error: code = NotFound desc = could not find container \"abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1\": container with ID starting with abbfa345dd59e3907cd0f9f718bfeb50a8901d10973b27d47a5f5ddad94f86d1 not found: ID does not exist"
Jul 01 21:09:55.492261 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.492190    2554 scope.go:117] "RemoveContainer" containerID="78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"
Jul 01 21:09:55.492444 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.492426    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301"} err="failed to get container status \"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301\": rpc error: code = NotFound desc = could not find container \"78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301\": container with ID starting with 78ddc4a5887af0fa3679446d6b82139a1d84fab91578dbd242ee9ed2bb587301 not found: ID does not exist"
Jul 01 21:09:55.492962 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:55.492928    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh"]
Jul 01 21:09:56.772332 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:09:56.772299    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" path="/var/lib/kubelet/pods/4d0ae410-7edd-4422-b989-2b422e80bae8/volumes"
Jul 01 21:10:32.550607 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:32.550569    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cbm9x_05f7d919-ddfa-438a-9dd8-e5c6d4ed8498/global-pull-secret-syncer/0.log"
Jul 01 21:10:32.721824 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:32.721794    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kdd8w_b26c1158-7a44-4ec3-afc7-f3b0c2c4a047/konnectivity-agent/0.log"
Jul 01 21:10:32.770116 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:32.770089    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-137.ec2.internal_0d191265e6bbab3f8d2d9679a92fcb37/haproxy/0.log"
Jul 01 21:10:36.908580 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:36.908545    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/alertmanager/0.log"
Jul 01 21:10:36.932189 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:36.932166    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/config-reloader/0.log"
Jul 01 21:10:36.962429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:36.962407    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/kube-rbac-proxy-web/0.log"
Jul 01 21:10:36.984985 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:36.984963    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/kube-rbac-proxy/0.log"
Jul 01 21:10:37.012971 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.012954    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/kube-rbac-proxy-metric/0.log"
Jul 01 21:10:37.036221 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.036196    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/prom-label-proxy/0.log"
Jul 01 21:10:37.058606 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.058585    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b86f04a3-572c-4a38-bfae-573a0d01835f/init-config-reloader/0.log"
Jul 01 21:10:37.218215 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.218142    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c78cb989c-7z9vp_31a7f6cb-d4c0-4cbc-adeb-ea37c920adae/metrics-server/0.log"
Jul 01 21:10:37.244734 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.244709    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-9fbqd_cfee7caf-006b-47f1-a417-274f5a11abfb/monitoring-plugin/0.log"
Jul 01 21:10:37.442342 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.442314    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hxdh_15a785b1-a9ad-42d3-b435-895abbfd3b3e/node-exporter/0.log"
Jul 01 21:10:37.466950 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.466913    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hxdh_15a785b1-a9ad-42d3-b435-895abbfd3b3e/kube-rbac-proxy/0.log"
Jul 01 21:10:37.490088 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.490008    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hxdh_15a785b1-a9ad-42d3-b435-895abbfd3b3e/init-textfile/0.log"
Jul 01 21:10:37.781618 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.781514    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-2gssg_01dcc36e-9a6d-4cec-b76d-3e9006849b7a/prometheus-operator/0.log"
Jul 01 21:10:37.800119 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.800093    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-2gssg_01dcc36e-9a6d-4cec-b76d-3e9006849b7a/kube-rbac-proxy/0.log"
Jul 01 21:10:37.827696 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:37.827668    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-kwzdl_598eb134-3f95-4bac-b5df-6242a4746e38/prometheus-operator-admission-webhook/0.log"
Jul 01 21:10:39.342396 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:39.342367    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-2rr8g_a2008ae7-08d6-4147-944b-492d48480330/networking-console-plugin/0.log"
Jul 01 21:10:39.821372 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:39.821342    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/1.log"
Jul 01 21:10:39.831540 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:39.831515    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-g4bw7_84f07d78-1d8e-48bf-abf6-3db7eae8ba51/console-operator/2.log"
Jul 01 21:10:40.962692 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.962651    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"]
Jul 01 21:10:40.963190 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963159    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container"
Jul 01 21:10:40.963190 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963177    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container"
Jul 01 21:10:40.963190 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963189    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kube-rbac-proxy"
Jul 01 21:10:40.963366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963198    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kube-rbac-proxy"
Jul 01 21:10:40.963366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963214    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="storage-initializer"
Jul 01 21:10:40.963366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963223    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="storage-initializer"
Jul 01 21:10:40.963366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963310    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kserve-container"
Jul 01 21:10:40.963366 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.963326    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0ae410-7edd-4422-b989-2b422e80bae8" containerName="kube-rbac-proxy"
Jul 01 21:10:40.966465 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.966444    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:40.969023 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.968995    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct4wg\"/\"openshift-service-ca.crt\""
Jul 01 21:10:40.969987 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.969970    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ct4wg\"/\"default-dockercfg-nqgvv\""
Jul 01 21:10:40.970054 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.969995    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ct4wg\"/\"kube-root-ca.crt\""
Jul 01 21:10:40.976429 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:40.976408    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"]
Jul 01 21:10:41.090055 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.090020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbrk\" (UniqueName: \"kubernetes.io/projected/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-kube-api-access-7wbrk\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.090232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.090083    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-lib-modules\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.090232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.090104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-podres\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.090232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.090125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-sys\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.090232 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.090157    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-proc\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.190874 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-proc\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190880    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbrk\" (UniqueName: \"kubernetes.io/projected/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-kube-api-access-7wbrk\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-lib-modules\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190972    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-podres\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190989    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-proc\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.190992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-sys\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191049 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.191035    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-sys\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191267 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.191122    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-podres\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.191267 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.191144    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-lib-modules\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.206689 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.206658    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbrk\" (UniqueName: \"kubernetes.io/projected/7925a1f4-3ba2-404c-adfc-95de2e7d82c7-kube-api-access-7wbrk\") pod \"perf-node-gather-daemonset-4jhqx\" (UID: \"7925a1f4-3ba2-404c-adfc-95de2e7d82c7\") " pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.276730 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.276655    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.403842 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.403814    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"]
Jul 01 21:10:41.405661 ip-10-0-129-137 kubenswrapper[2554]: W0701 21:10:41.405633    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7925a1f4_3ba2_404c_adfc_95de2e7d82c7.slice/crio-21c9aaf1ff2ed50d5cf2fc7ee2988ec2a8a783c8900b833d3f80cd1bcc838fab WatchSource:0}: Error finding container 21c9aaf1ff2ed50d5cf2fc7ee2988ec2a8a783c8900b833d3f80cd1bcc838fab: Status 404 returned error can't find the container with id 21c9aaf1ff2ed50d5cf2fc7ee2988ec2a8a783c8900b833d3f80cd1bcc838fab
Jul 01 21:10:41.672657 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.672615    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx" event={"ID":"7925a1f4-3ba2-404c-adfc-95de2e7d82c7","Type":"ContainerStarted","Data":"edde97f1a74eb0d7c45d5a640f5274d9116d4bc813de320b72a4ff953f197adf"}
Jul 01 21:10:41.672847 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.672670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx" event={"ID":"7925a1f4-3ba2-404c-adfc-95de2e7d82c7","Type":"ContainerStarted","Data":"21c9aaf1ff2ed50d5cf2fc7ee2988ec2a8a783c8900b833d3f80cd1bcc838fab"}
Jul 01 21:10:41.672847 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.672745    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:41.689084 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.689035    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx" podStartSLOduration=1.689019172 podStartE2EDuration="1.689019172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:10:41.687447619 +0000 UTC m=+1475.370931524" watchObservedRunningTime="2026-07-01 21:10:41.689019172 +0000 UTC m=+1475.372503075"
Jul 01 21:10:41.766482 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.766444    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzns2_f752519e-26b4-4130-b085-0b3b60b7a89e/dns/0.log"
Jul 01 21:10:41.788661 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.788634    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzns2_f752519e-26b4-4130-b085-0b3b60b7a89e/kube-rbac-proxy/0.log"
Jul 01 21:10:41.859037 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:41.859007    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qn9x_78922f2f-7ca3-468e-ae75-5a9cb7ca1563/dns-node-resolver/0.log"
Jul 01 21:10:42.384340 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:42.384309    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24krb_153889c1-36ed-424f-ad2e-4e3d98b84921/node-ca/0.log"
Jul 01 21:10:43.590103 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:43.590074    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dq69_6632a983-a19e-47ef-a1d5-ff80bf434faf/serve-healthcheck-canary/0.log"
Jul 01 21:10:43.995636 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:43.995608    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dxpb_e7e8ef8f-02f7-4167-b375-4af7ec26e660/kube-rbac-proxy/0.log"
Jul 01 21:10:44.016843 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:44.016813    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dxpb_e7e8ef8f-02f7-4167-b375-4af7ec26e660/exporter/0.log"
Jul 01 21:10:44.039287 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:44.039263    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dxpb_e7e8ef8f-02f7-4167-b375-4af7ec26e660/extractor/0.log"
Jul 01 21:10:46.192076 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.192048    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-67485586b-vgdj8_ce354ab3-79ac-4e24-9278-7a093e9ae9cc/manager/0.log"
Jul 01 21:10:46.235207 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.235177    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-2fkrf_2999b69c-49bd-49e6-903b-335d45ca6b5e/server/0.log"
Jul 01 21:10:46.339537 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.339501    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-w6lg2_0f5d24f6-47e0-4e36-a07a-4bdfe85dea30/manager/0.log"
Jul 01 21:10:46.380960 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.380918    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-d8lbp_52e8d727-d72e-48c7-803a-d4aaafa9421c/s3-init/0.log"
Jul 01 21:10:46.401986 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.401960    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-d8lbp_52e8d727-d72e-48c7-803a-d4aaafa9421c/download-hf-model/0.log"
Jul 01 21:10:46.434196 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.434169    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-fb4vr_c9f30b2f-0d09-4d7f-bb96-c33d02253348/s3-tls-init-custom/0.log"
Jul 01 21:10:46.451397 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.451334    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-fb4vr_c9f30b2f-0d09-4d7f-bb96-c33d02253348/download-hf-model/0.log"
Jul 01 21:10:46.485256 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.485211    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-7ltmz_f70df1be-8543-4e6b-8c48-cfc208d3bd68/s3-tls-init-serving/0.log"
Jul 01 21:10:46.503794 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.503773    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-7ltmz_f70df1be-8543-4e6b-8c48-cfc208d3bd68/download-hf-model/0.log"
Jul 01 21:10:46.556629 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.556596    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-tc2x5_571c213d-8814-4641-bf76-6abf06432878/seaweedfs-tls-custom/0.log"
Jul 01 21:10:46.579916 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:46.579887    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-zmhxj_2a9f84b1-170a-448c-8318-3c0b8597ec4d/seaweedfs-tls-serving/0.log"
Jul 01 21:10:47.688099 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:47.688072    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ct4wg/perf-node-gather-daemonset-4jhqx"
Jul 01 21:10:50.851377 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:50.851297    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-4zcjv_022a57d4-a679-49c2-98ac-2c6a12178ff8/kube-storage-version-migrator-operator/1.log"
Jul 01 21:10:50.852972 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:50.852928    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-4zcjv_022a57d4-a679-49c2-98ac-2c6a12178ff8/kube-storage-version-migrator-operator/0.log"
Jul 01 21:10:51.783026 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.782997    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/kube-multus-additional-cni-plugins/0.log"
Jul 01 21:10:51.804095 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.804069    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/egress-router-binary-copy/0.log"
Jul 01 21:10:51.824051 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.824023    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/cni-plugins/0.log"
Jul 01 21:10:51.843676 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.843654    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/bond-cni-plugin/0.log"
Jul 01 21:10:51.863652 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.863631    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/routeoverride-cni/0.log"
Jul 01 21:10:51.883473 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.883454    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/whereabouts-cni-bincopy/0.log"
Jul 01 21:10:51.902870 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:51.902850    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-chk96_0c6c57df-4530-4a13-9437-4256b6ccff0d/whereabouts-cni/0.log"
Jul 01 21:10:52.297000 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:52.296903    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q7xh8_95595c7a-7522-4735-9d27-2ac1237c8f03/kube-multus/0.log"
Jul 01 21:10:52.366798 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:52.366771    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7fwz5_8e9a83a3-4849-4baf-a9a7-781bf7974983/network-metrics-daemon/0.log"
Jul 01 21:10:52.385479 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:52.385449    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7fwz5_8e9a83a3-4849-4baf-a9a7-781bf7974983/kube-rbac-proxy/0.log"
Jul 01 21:10:53.155985 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.155953    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-controller/0.log"
Jul 01 21:10:53.171736 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.171703    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/0.log"
Jul 01 21:10:53.171884 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.171854    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovn-acl-logging/1.log"
Jul 01 21:10:53.195361 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.195333    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/kube-rbac-proxy-node/0.log"
Jul 01 21:10:53.217981 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.217949    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 21:10:53.235415 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.235390    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/northd/0.log"
Jul 01 21:10:53.255773 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.255738    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/nbdb/0.log"
Jul 01 21:10:53.278961 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.276428    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/sbdb/0.log"
Jul 01 21:10:53.346190 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:53.346155    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4m68j_9608f8a5-7473-4876-a9de-fd6b06753e13/ovnkube-controller/0.log"
Jul 01 21:10:54.671611 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:54.671579    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56499946bc-4bj29_182b1f4c-e9ad-4de8-86a9-fddcadd3952f/check-endpoints/0.log"
Jul 01 21:10:54.692991 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:54.692963    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9scdv_e4f34b82-ccc9-474f-8fd5-f332edbdacc6/network-check-target-container/0.log"
Jul 01 21:10:55.632061 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:55.632021    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wqq4l_65b00689-c10f-4b94-a41e-292d4ed16225/iptables-alerter/0.log"
Jul 01 21:10:56.191339 ip-10-0-129-137 kubenswrapper[2554]: I0701 21:10:56.191309    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-qbsrl_add06c46-3ee3-4b0b-b239-9b2356dd099e/tuned/0.log"