Jul 07 18:58:44.540060 ip-10-0-140-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 18:58:44.540072 ip-10-0-140-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 18:58:44.540079 ip-10-0-140-56 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 18:58:44.540330 ip-10-0-140-56 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 18:58:54.759302 ip-10-0-140-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 18:58:54.759317 ip-10-0-140-56 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c602d5cb2f294a7c80bb7226c0d4b2d9 --
Jul 07 19:00:32.788296 ip-10-0-140-56 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.036352    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053625    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053640    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053644    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053647    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053650    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053652    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053655    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.090158 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053658    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053661    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053663    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053666    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053669    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053671    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053674    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053676    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053679    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053681    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053684    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053686    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053689    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053691    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053693    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053696    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053718    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053722    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053725    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053727    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.092497 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053730    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053734    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053737    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053740    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053743    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053748    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053752    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053755    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053758    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053761    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053764    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053770    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053774    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053777    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053780    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053783    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053786    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053789    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053792    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.093556 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053794    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053797    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053800    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053802    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053805    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053807    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053810    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053812    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053815    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053818    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053820    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053822    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053825    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053829    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053831    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053834    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053836    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053839    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053842    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053845    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.095229 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053847    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053851    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053854    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053857    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053861    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053864    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053866    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053869    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053873    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053875    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053878    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053880    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053883    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053885    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053888    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053891    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053893    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053905    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053907    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.096100 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053910    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053912    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053915    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053918    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053920    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053923    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053926    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053929    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053932    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053934    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053937    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053939    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053942    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.053944    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054719    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054728    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054730    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054733    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054736    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054738    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.098078 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054741    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054744    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054746    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054749    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054751    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054754    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054757    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054760    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054762    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054765    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054768    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054777    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054779    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054782    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054785    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054790    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054794    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054797    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054799    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.098956 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054802    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054805    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054807    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054810    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054812    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054816    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054820    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054823    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054825    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054828    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054831    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054833    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054836    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054839    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054843    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054847    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054850    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054852    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054855    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.101471 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054858    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054860    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054863    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054865    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054868    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054871    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054880    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054882    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054885    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054888    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054891    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054893    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054896    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054898    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054901    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054903    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054906    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054908    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054911    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054913    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.102312 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054916    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054918    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054921    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054923    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054927    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054930    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054932    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054934    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054937    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054939    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054942    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054945    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054947    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054950    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054952    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054955    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054957    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054960    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054963    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054971    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.104658 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054973    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054977    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054980    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054982    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054985    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054987    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054990    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054992    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054995    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.054997    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055000    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055002    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055004    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055007    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055009    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055438    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055442    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055445    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055448    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055450    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.105516 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055455    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055458    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055460    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055463    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055466    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055469    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055471    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055474    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055477    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055479    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055482    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055491    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055494    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055496    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055499    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055502    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055505    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055507    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055510    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055512    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.108480 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055515    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055517    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055521    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055524    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055526    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055529    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055531    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055534    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055536    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055538    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055541    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055543    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055547    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055550    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055552    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055555    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055557    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055560    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055563    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055565    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.109342 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055568    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055570    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055572    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055575    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055583    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055587    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055591    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055595    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055597    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055600    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055603    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055605    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055608    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055611    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055614    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055617    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055619    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055622    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055624    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.112171 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055630    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055634    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055637    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055640    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055644    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055646    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055649    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055651    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055654    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055656    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055659    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055662    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055664    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055667    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055669    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055672    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055690    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055713    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.113037 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055718    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055725    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055728    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055731    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055734    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055737    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055741    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055744    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055748    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055751    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055754    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055757    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055759    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055762    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055765    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.055767    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055829    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055835    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055845    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055850    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 19:00:35.115418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055862    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055866    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055870    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055876    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055880    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055884    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055888    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055893    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055898    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055902    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055906    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055910    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055925    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055930    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055939    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055943    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055947    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055952    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055957    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055962    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055968    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055973    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055978    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055982    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 19:00:35.116306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055987    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055992    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.055996    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056001    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056006    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056011    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056018    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056022    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056026    2560 flags.go:64] FLAG: --enable-server="true"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056030    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056039    2560 flags.go:64] FLAG: --event-burst="100"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056044    2560 flags.go:64] FLAG: --event-qps="50"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056048    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056052    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056056    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056061    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056066    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056070    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056074    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056078    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056082    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056093    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056098    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056103    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056106    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 19:00:35.118671 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056111    2560 flags.go:64] FLAG: --feature-gates=""
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056116    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056120    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056127    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056131    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056136    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056140    2560 flags.go:64] FLAG: --help="false"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056144    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056149    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056153    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056157    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056162    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056167    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056172    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056176    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056180    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056184    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056189    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056193    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056197    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056202    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056206    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056210    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056214    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 19:00:35.119989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056218    2560 flags.go:64] FLAG: --lock-file=""
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056222    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056226    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056230    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056238    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056248    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056253    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056256    2560 flags.go:64] FLAG: --logging-format="text"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056261    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056265    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056270    2560 flags.go:64] FLAG: --manifest-url=""
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056275    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056281    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056286    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056291    2560 flags.go:64] FLAG: --max-pods="110"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056295    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056299    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056304    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056308    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056312    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056317    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056322    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056332    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056336    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056341    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 19:00:35.123270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056345    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056349    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056356    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056360    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056364    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056368    2560 flags.go:64] FLAG: --port="10250"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056372    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056376    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b66e79afd7d97a24"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056381    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056386    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056390    2560 flags.go:64] FLAG: --register-node="true"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056394    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056399    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056410    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056415    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056419    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056429    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056433    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056439    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056443    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056447    2560 flags.go:64] FLAG: --runonce="false"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056452    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056456    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056460    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056465    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 19:00:35.124392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056469    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056473    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056477    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056482    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056487    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056491    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056495    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056499    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056504    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056508    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056513    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056520    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056524    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056528    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056538    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056542    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056547    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056551    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056555    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056559    2560 flags.go:64] FLAG: --v="2"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056564    2560 flags.go:64] FLAG: --version="false"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056577    2560 flags.go:64] FLAG: --vmodule=""
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056583    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.056588    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 19:00:35.127672 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056789    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056800    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056806    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056810    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056814    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056818    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056821    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056826    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056830    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056833    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056837    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056841    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056848    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056853    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056857    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056861    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056865    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056868    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056872    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.265965 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056876    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.157970 ip-10-0-140-56 systemd[1]: Started Kubernetes Kubelet.
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056880    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056883    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056887    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056891    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056894    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056898    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056902    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056906    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056910    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056913    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056918    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056922    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056926    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056931    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056934    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056938    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056942    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056945    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056949    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.430223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056953    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056957    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056960    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056964    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056967    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056972    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056976    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056980    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056985    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056991    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056996    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.056999    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057003    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057007    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057010    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057014    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057018    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057021    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057025    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.431782 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057029    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057033    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057037    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057040    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057044    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057049    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057052    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057058    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057062    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057065    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057069    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057073    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057077    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057081    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057085    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057088    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057093    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057097    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057101    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057105    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.434616 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057109    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057112    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057116    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057120    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057124    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057128    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057131    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057135    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057139    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057143    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057146    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057150    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057154    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057158    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057162    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057166    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057169    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057174    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057178    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057184    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.435826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.057188    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.058724    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.068620    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.068634    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.070539    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.070550    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070603    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070609    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070612    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070616    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070619    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070621    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070624    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070627    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.439624 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070629    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070632    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070635    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070637    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070639    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070642    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070645    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070647    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070650    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070652    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070655    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070658    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070661    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070663    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070666    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070669    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070672    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070674    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070677    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070680    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.440562 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070683    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070686    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070689    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070691    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070695    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070712    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070717    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070721    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070725    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070728    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070731    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070734    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070737    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070740    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070743    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070745    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070748    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070751    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070753    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.445134 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070757    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070759    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070762    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070765    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070767    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070770    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070772    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070775    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070778    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070780    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070783    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070786    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070788    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070791    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070793    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070796    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070799    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070802    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070804    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070806    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.446318 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070810    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070813    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070817    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070821    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070824    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070829    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070831    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070834    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070836    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070839    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070842    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070844    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070848    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070851    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070853    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070856    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070858    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070861    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070863    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070866    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.447475 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070868    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070870    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070873    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070875    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070878    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070881    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070883    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070886    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070889    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070892    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070894    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.070897    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.070902    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071010    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:00:35.451355 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071015    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071018    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071021    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071024    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071028    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071031    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071034    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071036    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071039    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071042    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071046    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071049    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071051    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071054    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071056    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071059    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071061    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071064    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071066    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071069    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:00:35.452216 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071072    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071074    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071077    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071079    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071081    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071084    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071086    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071089    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071092    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071095    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071097    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071099    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071102    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071104    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071107    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071109    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071113    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071116    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071118    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071121    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:00:35.453304 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071123    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071126    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071129    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071132    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071134    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071137    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071140    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071142    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071145    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071147    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071150    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071152    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071155    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071157    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071160    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071162    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071165    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071167    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071170    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:00:35.455162 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071172    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071175    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071178    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071180    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071182    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071185    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071187    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071190    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071192    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071196    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071198    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071201    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071204    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071208    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071211    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071214    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071217    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071219    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071222    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:00:35.456032 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071225    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071227    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071229    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071232    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071234    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071237    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071240    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071242    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071244    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071249    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071252    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071255    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071258    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071261    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071263    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071267    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071269    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071272    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071274    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:00:35.458144 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:35.071276    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.071281    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.071376    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.075255    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.078775    2560 server.go:1019] "Starting client certificate rotation"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.078871    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.078906    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.086116    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.088458    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.105926    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.110713    2560 log.go:25] "Validated CRI v1 image API"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.111363    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 19:00:35.459020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.113601    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 19:00:35.618100 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.118813    2560 fs.go:135] Filesystem UUIDs: map[13f35bd5-3264-4ade-a9fe-0ef317b111d8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e46e849b-f0ff-4a0f-b41b-bd0af2a04075:/dev/nvme0n1p3]
Jul 07 19:00:35.618100 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.118826    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.124111    2560 manager.go:216] Machine: {Timestamp:2026-07-07 19:00:35.12180095 +0000 UTC m=+0.841600132 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097765 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2124ce585899264d24adfe3010d218 SystemUUID:ec2124ce-5858-9926-4d24-adfe3010d218 BootID:c602d5cb-2f29-4a7c-80bb-7226c0d4b2d9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:70:f6:55:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:70:f6:55:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:87:56:48:09:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.124207    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.124273    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.131015    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.131032    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.132632    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.132642    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.132654    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.134227    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.138486    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144760    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144876    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144901    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144910    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144926    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144938    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.144946    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 19:00:35.783384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.145246    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.145261    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.148077    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.150086    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.150113    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151345    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151363    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151370    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151377    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151389    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151396    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151406    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151411    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151418    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151423    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151431    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.151436    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.152620    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.152794    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.157272    2560 server.go:1268] "Started kubelet"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.157651    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.158500    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.158590    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.158755    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.163140    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.163222    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.163494    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.163772    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.176782    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.176575    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-56.ec2.internal.18c01715dfebfa7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-56.ec2.internal,UID:ip-10-0-140-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-56.ec2.internal,},FirstTimestamp:2026-07-07 19:00:35.157252732 +0000 UTC m=+0.877051954,LastTimestamp:2026-07-07 19:00:35.157252732 +0000 UTC m=+0.877051954,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-56.ec2.internal,}"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.180623    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-62wnc"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.192470    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.192498    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-62wnc"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.192499    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.192585    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193481    2560 factory.go:55] Registering systemd factory
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193493    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193689    2560 factory.go:153] Registering CRI-O factory
Jul 07 19:00:35.784437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193711    2560 factory.go:223] Registration of the crio container factory successfully
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193751    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193769    2560 factory.go:103] Registering Raw factory
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.193783    2560 manager.go:1196] Started watching for new ooms in manager
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194049    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194065    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194047    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194157    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194163    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.194415    2560 manager.go:349] Starting recovery of all containers
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.207310    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.209142    2560 manager.go:354] Recovery completed
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.211841    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-56.ec2.internal\" not found" node="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.213124    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.214779    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.214804    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.214813    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.215276    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.215285    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.215299    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.215326    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216335    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216342    2560 policy_none.go:49] "None policy: Start"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216357    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216363    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216370    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.216378    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.217339    2560 policy_none.go:47] "Start"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274240    2560 manager.go:337] "Starting Device Plugin manager"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.274368    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274379    2560 server.go:87] "Starting device plugin registration server"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274557    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274565    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274664    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274774    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.274783    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.275195    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.275234    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.374838    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.375587    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.375610    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.375620    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.375642    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.392591    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.392607    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-56.ec2.internal\": node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.400667    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.401729    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.401758    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.401775    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.401804    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 19:00:35.785989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.406635    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.424441    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.502371    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"]
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.503272    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.504082    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.504113    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.504129    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.505589    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.505716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.505751    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506259    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506274    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506295    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506304    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506281    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.506363    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.507429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.507452    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.512947    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.512969    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.512984    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.524949    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.530188    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-56.ec2.internal\" not found" node="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.533416    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-56.ec2.internal\" not found" node="ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.596251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.596273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.596288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e764c2d22d220cd20bf06ff00832f0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-56.ec2.internal\" (UID: \"e764c2d22d220cd20bf06ff00832f0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.625290    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.788548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696544    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.790058 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.790058 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696586    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/922e395221ca99f0b2c4a3815455d230-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal\" (UID: \"922e395221ca99f0b2c4a3815455d230\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.790058 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e764c2d22d220cd20bf06ff00832f0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-56.ec2.internal\" (UID: \"e764c2d22d220cd20bf06ff00832f0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.790058 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.696636    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e764c2d22d220cd20bf06ff00832f0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-140-56.ec2.internal\" (UID: \"e764c2d22d220cd20bf06ff00832f0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.790058 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.725996    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.826565 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.826551    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.833707 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.833692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.836378 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.836361    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:35.927515 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:35.927492    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-56.ec2.internal\" not found"
Jul 07 19:00:35.987411 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.987390    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:35.993603 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:35.993588    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:36.011313 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.011295    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 19:00:36.013542 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.013530    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal"
Jul 07 19:00:36.030317 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.030298    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 19:00:36.076452 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.076399    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 19:00:36.076525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.076508    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 19:00:36.076562 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.076531    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 19:00:36.076562 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.076535    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 19:00:36.145714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.145685    2560 apiserver.go:52] "Watching apiserver"
Jul 07 19:00:36.158430 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.158411    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 19:00:36.159642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.159621    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n","openshift-multus/multus-additional-cni-plugins-xxnjb","openshift-multus/network-metrics-daemon-pskjn","openshift-network-diagnostics/network-check-target-xdrgp","openshift-network-operator/iptables-alerter-cxzgf","openshift-ovn-kubernetes/ovnkube-node-c28xf","kube-system/global-pull-secret-syncer-klwmk","kube-system/konnectivity-agent-n54g5","openshift-cluster-node-tuning-operator/tuned-g7772","openshift-dns/node-resolver-lp9zc","openshift-image-registry/node-ca-m64kp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal","openshift-multus/multus-t65ms"]
Jul 07 19:00:36.161353 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.161329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.162632 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.162613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.163662 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.163640    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.163767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.163687    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 19:00:36.163767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.163720    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mqmcg\""
Jul 07 19:00:36.163767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.163743    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.163910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.163721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 19:00:36.165136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.164622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.165136 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.164733    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:36.165136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.164869    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 19:00:36.165136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.164929    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 19:00:36.165370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.165202    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.165370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.165247    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hcwcs\""
Jul 07 19:00:36.165472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.165454    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 19:00:36.165548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.165532    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 19:00:36.165948 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.165931    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.166344 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.166169    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 19:00:36.166792 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.166453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:36.166792 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.166524    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:36.167837 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.167820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.169068 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.169052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.169354 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.169336    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.169518 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.169505    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.169518 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.169513    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 19:00:36.169734 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.169720    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xzqls\""
Jul 07 19:00:36.170338 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.170322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.170861 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.170845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 19:00:36.170926 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.170914    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nfmlp\""
Jul 07 19:00:36.170979 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.170927    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 19:00:36.171934 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.171918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.172020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.171990    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqr72\""
Jul 07 19:00:36.172121 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.171990    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 19:00:36.173467 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.173453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v5zg\""
Jul 07 19:00:36.173659 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.173645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.173731 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.173658    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.174076 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.174062    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 19:00:36.174242 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.174227    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.174895 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.174882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.175264 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.175249    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.175400 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.175390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-spjkw\""
Jul 07 19:00:36.175564 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.175553    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.176206 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.176191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.176517 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.176502    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.176618 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.176604    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.176733 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.176717    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95q2m\""
Jul 07 19:00:36.177772 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.177759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.177823 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.177809    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:36.178247 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.178229    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g5td5\""
Jul 07 19:00:36.178455 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.178440    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 19:00:36.178562 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.178544    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 19:00:36.178643 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.178593    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 19:00:36.192988 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.192971    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 19:00:36.195796 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.195781    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 19:00:36.197110 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.197065    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:55:35 +0000 UTC" deadline="2028-01-31 14:47:21.753302217 +0000 UTC"
Jul 07 19:00:36.197183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.197110    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13747h46m45.556196185s"
Jul 07 19:00:36.200027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.199912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e985872-3005-4741-95a2-c57a7335b8bc-ovn-node-metrics-cert\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.199932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cnibin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.199954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-netns\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.199999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysconfig\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-etc-tuned\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-script-lib\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqb8\" (UniqueName: \"kubernetes.io/projected/23595584-cbfb-4ad5-985a-449f84e148c0-kube-api-access-hjqb8\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-netns\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-var-lib-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200240 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-node-log\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-log-socket\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-run\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-tmp-dir\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-slash\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-multus-certs\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-etc-kubernetes\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-lib-modules\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49533686-cfa9-4fdb-80c5-38fd831c76ed-iptables-alerter-script\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-k8s-cni-cncf-io\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-daemon-config\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-tmp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-systemd-units\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-system-cni-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkqr\" (UniqueName: \"kubernetes.io/projected/dddb970a-248c-4fb2-aca6-49698d3f75da-kube-api-access-rtkqr\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-sys-fs\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-os-release\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-multus\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-systemd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-bin\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9vt\" (UniqueName: \"kubernetes.io/projected/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-kube-api-access-7r9vt\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-kubelet\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-systemd\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.200962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-var-lib-kubelet\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-konnectivity-ca\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-cnibin\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-ovn\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.200986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5g6\" (UniqueName: \"kubernetes.io/projected/5ec87123-2d72-49e9-987c-195a580ebf84-kube-api-access-tl5g6\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-conf\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-os-release\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49533686-cfa9-4fdb-80c5-38fd831c76ed-host-slash\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kcx\" (UniqueName: \"kubernetes.io/projected/49533686-cfa9-4fdb-80c5-38fd831c76ed-kube-api-access-q7kcx\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-agent-certs\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-hosts-file\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-host\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cni-binary-copy\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z967\" (UniqueName: \"kubernetes.io/projected/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-kube-api-access-6z967\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.201431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-bin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-hostroot\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-device-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dnfp\" (UniqueName: \"kubernetes.io/projected/3f401015-236d-44b9-a7ee-f1121723e831-kube-api-access-9dnfp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-kubelet-config\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-conf-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-registration-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-modprobe-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-netd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmcv\" (UniqueName: \"kubernetes.io/projected/9e985872-3005-4741-95a2-c57a7335b8bc-kube-api-access-cbmcv\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbm5n\" (UniqueName: \"kubernetes.io/projected/b75be16b-9497-4bfe-8e37-9dbffc5ce222-kube-api-access-xbm5n\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-kubernetes\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23595584-cbfb-4ad5-985a-449f84e148c0-host\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-kubelet\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-system-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-sys\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23595584-cbfb-4ad5-985a-449f84e148c0-serviceca\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-etc-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.201600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-socket-dir-parent\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.202100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-socket-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.202128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.202151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-dbus\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.202173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-config\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.202620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.202195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-env-overrides\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.210201 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.210184    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 19:00:36.241683 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.241665    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hl7n"
Jul 07 19:00:36.253517 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.253503    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hl7n"
Jul 07 19:00:36.302520 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kcx\" (UniqueName: \"kubernetes.io/projected/49533686-cfa9-4fdb-80c5-38fd831c76ed-kube-api-access-q7kcx\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-agent-certs\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-hosts-file\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-host\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cni-binary-copy\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z967\" (UniqueName: \"kubernetes.io/projected/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-kube-api-access-6z967\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.302663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-bin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302679    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-hosts-file\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302755    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-hostroot\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-hostroot\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-device-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dnfp\" (UniqueName: \"kubernetes.io/projected/3f401015-236d-44b9-a7ee-f1121723e831-kube-api-access-9dnfp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302836    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-host\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-kubelet-config\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302879    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302892    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-device-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302946    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-kubelet-config\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302957    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-bin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302969    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.302999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-conf-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303030    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-conf-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-registration-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-modprobe-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-netd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303164    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-registration-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmcv\" (UniqueName: \"kubernetes.io/projected/9e985872-3005-4741-95a2-c57a7335b8bc-kube-api-access-cbmcv\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303188    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303210    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-modprobe-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303229    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-netd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbm5n\" (UniqueName: \"kubernetes.io/projected/b75be16b-9497-4bfe-8e37-9dbffc5ce222-kube-api-access-xbm5n\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-kubernetes\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23595584-cbfb-4ad5-985a-449f84e148c0-host\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-kubelet\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303331    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cni-binary-copy\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303364    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.303795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303381    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23595584-cbfb-4ad5-985a-449f84e148c0-host\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303395    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-kubelet\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-system-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303398    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-kubernetes\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303436    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-system-cni-dir\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-sys\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23595584-cbfb-4ad5-985a-449f84e148c0-serviceca\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303482    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-etc-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303504    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-sys\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-socket-dir-parent\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303554    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-etc-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303558    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-socket-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.304625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-dbus\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303625    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-socket-dir-parent\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303660    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-socket-dir\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303675    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/443ebe9b-7063-4620-9025-3884c0734c8c-dbus\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-config\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303725    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-env-overrides\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e985872-3005-4741-95a2-c57a7335b8bc-ovn-node-metrics-cert\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cnibin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-netns\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysconfig\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303863    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-cnibin\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303875    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysconfig\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303918    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-netns\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-etc-tuned\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303962    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23595584-cbfb-4ad5-985a-449f84e148c0-serviceca\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-script-lib\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.305521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.303993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqb8\" (UniqueName: \"kubernetes.io/projected/23595584-cbfb-4ad5-985a-449f84e148c0-kube-api-access-hjqb8\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-netns\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-var-lib-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-node-log\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-log-socket\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-run\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304176    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-env-overrides\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-tmp-dir\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304234    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-var-lib-openvswitch\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304246    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-config\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.304347    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.304449    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:36.804408887 +0000 UTC m=+2.524208282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304785    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304822    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-run-netns\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.306406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304885    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-log-socket\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.304927    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-node-log\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305023    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-d\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305076    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-run\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305100    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-tmp-dir\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-slash\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-multus-certs\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-etc-kubernetes\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.305258    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305260    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e985872-3005-4741-95a2-c57a7335b8bc-ovnkube-script-lib\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-lib-modules\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49533686-cfa9-4fdb-80c5-38fd831c76ed-iptables-alerter-script\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305322    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-slash\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305361    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-etc-kubernetes\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-k8s-cni-cncf-io\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.307408 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305405    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-daemon-config\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305449    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-multus-certs\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-tmp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-systemd-units\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-system-cni-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkqr\" (UniqueName: \"kubernetes.io/projected/dddb970a-248c-4fb2-aca6-49698d3f75da-kube-api-access-rtkqr\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-sys-fs\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-os-release\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-multus\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-systemd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-bin\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9vt\" (UniqueName: \"kubernetes.io/projected/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-kube-api-access-7r9vt\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-kubelet\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-systemd\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-var-lib-kubelet\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-konnectivity-ca\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.308187 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-cnibin\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-ovn\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305882    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49533686-cfa9-4fdb-80c5-38fd831c76ed-iptables-alerter-script\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5g6\" (UniqueName: \"kubernetes.io/projected/5ec87123-2d72-49e9-987c-195a580ebf84-kube-api-access-tl5g6\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-conf\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-os-release\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305984    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-lib-modules\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.305989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49533686-cfa9-4fdb-80c5-38fd831c76ed-host-slash\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306029    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49533686-cfa9-4fdb-80c5-38fd831c76ed-host-slash\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306070    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-systemd-units\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306081    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-kubelet\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306127    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-system-cni-dir\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306130    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-var-lib-kubelet\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306187    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-cnibin\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306231    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-ovn\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306350    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ec87123-2d72-49e9-987c-195a580ebf84-sys-fs\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.308791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306407    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-os-release\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306441    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-var-lib-cni-multus\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306476    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-run-systemd\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306509    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e985872-3005-4741-95a2-c57a7335b8bc-host-cni-bin\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306606    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306713    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-sysctl-conf\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306735    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f401015-236d-44b9-a7ee-f1121723e831-etc-systemd\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306785    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e985872-3005-4741-95a2-c57a7335b8bc-ovn-node-metrics-cert\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306792    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-etc-tuned\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.306853    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddb970a-248c-4fb2-aca6-49698d3f75da-os-release\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.306999    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:00:36.806981057 +0000 UTC m=+2.526780231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.307303    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dddb970a-248c-4fb2-aca6-49698d3f75da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.307191    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-agent-certs\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.307194    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b852f3ea-6aa4-4e1e-82db-92562d7b5fa4-konnectivity-ca\") pod \"konnectivity-agent-n54g5\" (UID: \"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4\") " pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.307043    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b75be16b-9497-4bfe-8e37-9dbffc5ce222-host-run-k8s-cni-cncf-io\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.308070    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b75be16b-9497-4bfe-8e37-9dbffc5ce222-multus-daemon-config\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.309410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.308531    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f401015-236d-44b9-a7ee-f1121723e831-tmp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.312826 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.312806    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dnfp\" (UniqueName: \"kubernetes.io/projected/3f401015-236d-44b9-a7ee-f1121723e831-kube-api-access-9dnfp\") pod \"tuned-g7772\" (UID: \"3f401015-236d-44b9-a7ee-f1121723e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.313957 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.313934    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbm5n\" (UniqueName: \"kubernetes.io/projected/b75be16b-9497-4bfe-8e37-9dbffc5ce222-kube-api-access-xbm5n\") pod \"multus-t65ms\" (UID: \"b75be16b-9497-4bfe-8e37-9dbffc5ce222\") " pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.315176 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.315154    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqb8\" (UniqueName: \"kubernetes.io/projected/23595584-cbfb-4ad5-985a-449f84e148c0-kube-api-access-hjqb8\") pod \"node-ca-m64kp\" (UID: \"23595584-cbfb-4ad5-985a-449f84e148c0\") " pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.315269 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.315242    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmcv\" (UniqueName: \"kubernetes.io/projected/9e985872-3005-4741-95a2-c57a7335b8bc-kube-api-access-cbmcv\") pod \"ovnkube-node-c28xf\" (UID: \"9e985872-3005-4741-95a2-c57a7335b8bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.315362 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.315342    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kcx\" (UniqueName: \"kubernetes.io/projected/49533686-cfa9-4fdb-80c5-38fd831c76ed-kube-api-access-q7kcx\") pod \"iptables-alerter-cxzgf\" (UID: \"49533686-cfa9-4fdb-80c5-38fd831c76ed\") " pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.315696 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.315673    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z967\" (UniqueName: \"kubernetes.io/projected/6c2bd74c-a302-4fd8-94d0-c41ed8537fed-kube-api-access-6z967\") pod \"node-resolver-lp9zc\" (UID: \"6c2bd74c-a302-4fd8-94d0-c41ed8537fed\") " pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.316385 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.316369    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5g6\" (UniqueName: \"kubernetes.io/projected/5ec87123-2d72-49e9-987c-195a580ebf84-kube-api-access-tl5g6\") pod \"aws-ebs-csi-driver-node-h565n\" (UID: \"5ec87123-2d72-49e9-987c-195a580ebf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.316919 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.316905    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkqr\" (UniqueName: \"kubernetes.io/projected/dddb970a-248c-4fb2-aca6-49698d3f75da-kube-api-access-rtkqr\") pod \"multus-additional-cni-plugins-xxnjb\" (UID: \"dddb970a-248c-4fb2-aca6-49698d3f75da\") " pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.317047 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.317034    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:36.317083 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.317053    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:36.317083 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.317062    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:36.317146 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.317105    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:36.817092105 +0000 UTC m=+2.536891296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:36.318770 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.318754    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9vt\" (UniqueName: \"kubernetes.io/projected/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-kube-api-access-7r9vt\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.325966 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.325950    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g7772"
Jul 07 19:00:36.339211 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.339194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lp9zc"
Jul 07 19:00:36.342869 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.342853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m64kp"
Jul 07 19:00:36.357249 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.357234    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:36.501104 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.501078    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:00:36.522930 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.522911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxnjb"
Jul 07 19:00:36.538694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.538673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxzgf"
Jul 07 19:00:36.549657 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.549623    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49533686_cfa9_4fdb_80c5_38fd831c76ed.slice/crio-82fe31595a8d61a37a008f205eba3e03abd5abf12e40cd615959a19570d21f18 WatchSource:0}: Error finding container 82fe31595a8d61a37a008f205eba3e03abd5abf12e40cd615959a19570d21f18: Status 404 returned error can't find the container with id 82fe31595a8d61a37a008f205eba3e03abd5abf12e40cd615959a19570d21f18
Jul 07 19:00:36.550098 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.550083    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddb970a_248c_4fb2_aca6_49698d3f75da.slice/crio-09ec98ea4ddfb62c2a6249c175d289fd02ec80c6c93fa628d50a945514911bd3 WatchSource:0}: Error finding container 09ec98ea4ddfb62c2a6249c175d289fd02ec80c6c93fa628d50a945514911bd3: Status 404 returned error can't find the container with id 09ec98ea4ddfb62c2a6249c175d289fd02ec80c6c93fa628d50a945514911bd3
Jul 07 19:00:36.551620 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.551598    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode764c2d22d220cd20bf06ff00832f0cd.slice/crio-9b398a74fbc6e77d304085a8e675be3208ceaab366ca04424c76773dd6fd66ad WatchSource:0}: Error finding container 9b398a74fbc6e77d304085a8e675be3208ceaab366ca04424c76773dd6fd66ad: Status 404 returned error can't find the container with id 9b398a74fbc6e77d304085a8e675be3208ceaab366ca04424c76773dd6fd66ad
Jul 07 19:00:36.552221 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.552090    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922e395221ca99f0b2c4a3815455d230.slice/crio-12bb97bc15f75c63066073ca02cf0f4176f047df38a2b375e1a0d4e23f854818 WatchSource:0}: Error finding container 12bb97bc15f75c63066073ca02cf0f4176f047df38a2b375e1a0d4e23f854818: Status 404 returned error can't find the container with id 12bb97bc15f75c63066073ca02cf0f4176f047df38a2b375e1a0d4e23f854818
Jul 07 19:00:36.554014 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.553999    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:00:36.557603 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.557588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:36.564917 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.564900    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb852f3ea_6aa4_4e1e_82db_92562d7b5fa4.slice/crio-eaf225376a21542f3b20ec5b2cc1dcf2bc934ff09ef93462cebd9eaa87b83aab WatchSource:0}: Error finding container eaf225376a21542f3b20ec5b2cc1dcf2bc934ff09ef93462cebd9eaa87b83aab: Status 404 returned error can't find the container with id eaf225376a21542f3b20ec5b2cc1dcf2bc934ff09ef93462cebd9eaa87b83aab
Jul 07 19:00:36.571917 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.571899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t65ms"
Jul 07 19:00:36.578141 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.578122    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75be16b_9497_4bfe_8e37_9dbffc5ce222.slice/crio-4d2126c8c1f67baa9cab95dd5acb28625e131db1409343ac1dc211dc2f492707 WatchSource:0}: Error finding container 4d2126c8c1f67baa9cab95dd5acb28625e131db1409343ac1dc211dc2f492707: Status 404 returned error can't find the container with id 4d2126c8c1f67baa9cab95dd5acb28625e131db1409343ac1dc211dc2f492707
Jul 07 19:00:36.592800 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.592780    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n"
Jul 07 19:00:36.599438 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.599420    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec87123_2d72_49e9_987c_195a580ebf84.slice/crio-4357c544db67a8a1a822fc00bd1d9d31c05af56a48e69fe6aae87c51b6725041 WatchSource:0}: Error finding container 4357c544db67a8a1a822fc00bd1d9d31c05af56a48e69fe6aae87c51b6725041: Status 404 returned error can't find the container with id 4357c544db67a8a1a822fc00bd1d9d31c05af56a48e69fe6aae87c51b6725041
Jul 07 19:00:36.748207 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.748184    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2bd74c_a302_4fd8_94d0_c41ed8537fed.slice/crio-eb056a821696b9af5e07ff5308529868f52dd2256d047b13aba9fea4efd2d389 WatchSource:0}: Error finding container eb056a821696b9af5e07ff5308529868f52dd2256d047b13aba9fea4efd2d389: Status 404 returned error can't find the container with id eb056a821696b9af5e07ff5308529868f52dd2256d047b13aba9fea4efd2d389
Jul 07 19:00:36.798154 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.798131    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23595584_cbfb_4ad5_985a_449f84e148c0.slice/crio-59f5f71d214ca62a44889148b2e30828aa280ee248c4ecba271a8b0ae3564fa3 WatchSource:0}: Error finding container 59f5f71d214ca62a44889148b2e30828aa280ee248c4ecba271a8b0ae3564fa3: Status 404 returned error can't find the container with id 59f5f71d214ca62a44889148b2e30828aa280ee248c4ecba271a8b0ae3564fa3
Jul 07 19:00:36.811023 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.811005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:36.811096 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.811033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:36.811183 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.811168    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:36.811217 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.811202    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:36.811252 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.811221    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:00:37.811205068 +0000 UTC m=+3.531004236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:36.811252 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.811240    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:37.81122934 +0000 UTC m=+3.531028512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:36.816935 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.816916    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e985872_3005_4741_95a2_c57a7335b8bc.slice/crio-daae980bed42f19e33c6834daad6d5f9cf3c554861c0ce98994aa324a8bd426e WatchSource:0}: Error finding container daae980bed42f19e33c6834daad6d5f9cf3c554861c0ce98994aa324a8bd426e: Status 404 returned error can't find the container with id daae980bed42f19e33c6834daad6d5f9cf3c554861c0ce98994aa324a8bd426e
Jul 07 19:00:36.912116 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.912068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:36.912223 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.912208    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:36.912265 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.912227    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:36.912265 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.912237    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:36.912335 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:36.912277    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:37.912265161 +0000 UTC m=+3.632064328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:36.927192 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:36.927173    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:36.972986 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:00:36.972968    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f401015_236d_44b9_a7ee_f1121723e831.slice/crio-b08caa399913a8e02b9ac2357302de15fb18edbab82b42e80954c6b5e1bd3c60 WatchSource:0}: Error finding container b08caa399913a8e02b9ac2357302de15fb18edbab82b42e80954c6b5e1bd3c60: Status 404 returned error can't find the container with id b08caa399913a8e02b9ac2357302de15fb18edbab82b42e80954c6b5e1bd3c60
Jul 07 19:00:37.256095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.256017    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:36 +0000 UTC" deadline="2028-02-19 16:22:00.056026266 +0000 UTC"
Jul 07 19:00:37.256095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.256052    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14205h21m22.799978397s"
Jul 07 19:00:37.402292 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.402257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:37.402433 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.402394    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:37.412040 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.411020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n54g5" event={"ID":"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4","Type":"ContainerStarted","Data":"eaf225376a21542f3b20ec5b2cc1dcf2bc934ff09ef93462cebd9eaa87b83aab"}
Jul 07 19:00:37.416194 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.416168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxzgf" event={"ID":"49533686-cfa9-4fdb-80c5-38fd831c76ed","Type":"ContainerStarted","Data":"82fe31595a8d61a37a008f205eba3e03abd5abf12e40cd615959a19570d21f18"}
Jul 07 19:00:37.417464 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.417439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal" event={"ID":"922e395221ca99f0b2c4a3815455d230","Type":"ContainerStarted","Data":"12bb97bc15f75c63066073ca02cf0f4176f047df38a2b375e1a0d4e23f854818"}
Jul 07 19:00:37.420804 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.420773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerStarted","Data":"09ec98ea4ddfb62c2a6249c175d289fd02ec80c6c93fa628d50a945514911bd3"}
Jul 07 19:00:37.429177 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.429152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"daae980bed42f19e33c6834daad6d5f9cf3c554861c0ce98994aa324a8bd426e"}
Jul 07 19:00:37.430804 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.430782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m64kp" event={"ID":"23595584-cbfb-4ad5-985a-449f84e148c0","Type":"ContainerStarted","Data":"59f5f71d214ca62a44889148b2e30828aa280ee248c4ecba271a8b0ae3564fa3"}
Jul 07 19:00:37.435068 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.435044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lp9zc" event={"ID":"6c2bd74c-a302-4fd8-94d0-c41ed8537fed","Type":"ContainerStarted","Data":"eb056a821696b9af5e07ff5308529868f52dd2256d047b13aba9fea4efd2d389"}
Jul 07 19:00:37.439100 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.439076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n" event={"ID":"5ec87123-2d72-49e9-987c-195a580ebf84","Type":"ContainerStarted","Data":"4357c544db67a8a1a822fc00bd1d9d31c05af56a48e69fe6aae87c51b6725041"}
Jul 07 19:00:37.441055 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.441031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t65ms" event={"ID":"b75be16b-9497-4bfe-8e37-9dbffc5ce222","Type":"ContainerStarted","Data":"4d2126c8c1f67baa9cab95dd5acb28625e131db1409343ac1dc211dc2f492707"}
Jul 07 19:00:37.442691 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.442670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal" event={"ID":"e764c2d22d220cd20bf06ff00832f0cd","Type":"ContainerStarted","Data":"9b398a74fbc6e77d304085a8e675be3208ceaab366ca04424c76773dd6fd66ad"}
Jul 07 19:00:37.446019 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.445986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g7772" event={"ID":"3f401015-236d-44b9-a7ee-f1121723e831","Type":"ContainerStarted","Data":"b08caa399913a8e02b9ac2357302de15fb18edbab82b42e80954c6b5e1bd3c60"}
Jul 07 19:00:37.637389 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.637321    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:37.819402 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.819360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:37.819569 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.819412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:37.819569 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.819529    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:37.819676 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.819589    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:00:39.819569626 +0000 UTC m=+5.539368812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:37.820061 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.820039    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:37.820171 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.820097    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:39.820080167 +0000 UTC m=+5.539879355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:37.920601 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:37.920530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:37.920791 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.920773    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:37.920877 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.920798    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:37.920877 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.920811    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:37.920877 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:37.920864    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:39.920847245 +0000 UTC m=+5.640646414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:38.114944 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:38.114919    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:00:38.256431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:38.256313    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:36 +0000 UTC" deadline="2028-01-21 12:32:17.016135369 +0000 UTC"
Jul 07 19:00:38.256431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:38.256365    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13505h31m38.759773742s"
Jul 07 19:00:38.402930 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:38.402903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:38.403094 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:38.403022    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:38.403412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:38.403395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:38.403517 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:38.403498    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:39.405574 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:39.405066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:39.405574 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.405196    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:39.833485 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:39.833408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:39.833485 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:39.833455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:39.833695 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.833577    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:39.833695 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.833578    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:39.833695 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.833637    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:00:43.833618503 +0000 UTC m=+9.553417691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:39.833695 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.833654    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:43.833645197 +0000 UTC m=+9.553444372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:39.933830 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:39.933795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:39.933997 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.933976    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:39.934071 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.934000    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:39.934071 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.934014    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:39.934174 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:39.934077    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:43.934051244 +0000 UTC m=+9.653850424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:40.403072 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:40.403041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:40.403242 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:40.403190    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:40.403617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:40.403565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:40.403754 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:40.403652    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:41.402508 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:41.402052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:41.402508 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:41.402171    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:42.402009 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:42.401976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:42.402982 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:42.402944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:42.403689 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:42.403230    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:42.403998 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:42.403894    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:43.402345 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:43.402314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:43.402525 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.402456    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:43.864756 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:43.864712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:43.865167 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:43.864769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:43.865167 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.864836    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:43.865167 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.864908    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:51.864885992 +0000 UTC m=+17.584685162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:43.865167 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.864927    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:43.865167 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.864985    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:00:51.864966329 +0000 UTC m=+17.584765500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:43.965377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:43.965344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:43.965546 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.965492    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:43.965546 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.965526    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:43.965546 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.965538    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:43.965751 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:43.965591    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:51.965573324 +0000 UTC m=+17.685372512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:44.403451 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:44.402859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:44.403451 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:44.402984    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:44.403451 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:44.403046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:44.403451 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:44.403132    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:45.403931 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:45.403901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:45.404383 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:45.404008    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:46.402647 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:46.402612    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:46.402838 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:46.402764    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:46.403191 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:46.403160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:46.403303 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:46.403275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:47.402497 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:47.402352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:47.402913 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:47.402518    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:48.402820 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.402578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:48.403235 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.402578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:48.403235 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:48.402880    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:48.403235 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:48.402947    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:48.473436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.473404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal" event={"ID":"e764c2d22d220cd20bf06ff00832f0cd","Type":"ContainerStarted","Data":"905d689e7bab4442086b2e46ae3cc18832d54870ad5ebec26b44afb7eae9c7c9"}
Jul 07 19:00:48.475286 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.475261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g7772" event={"ID":"3f401015-236d-44b9-a7ee-f1121723e831","Type":"ContainerStarted","Data":"fa400b380c071a5d7a9e418abeaf338992939c1321c7124fb56d0900697134a4"}
Jul 07 19:00:48.492896 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.492852    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-56.ec2.internal" podStartSLOduration=12.492840431 podStartE2EDuration="12.492840431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:48.492754464 +0000 UTC m=+14.212553652" watchObservedRunningTime="2026-07-07 19:00:48.492840431 +0000 UTC m=+14.212639620"
Jul 07 19:00:48.512155 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:48.512118    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g7772" podStartSLOduration=3.080760347 podStartE2EDuration="13.512106791s" totalImagesPullingTime="10.431346444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:48.51190778 +0000 UTC m=+14.231706972" watchObservedRunningTime="2026-07-07 19:00:48.512106791 +0000 UTC m=+14.231905981"
Jul 07 19:00:49.402460 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.402380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:49.402636 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:49.402512    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:49.479344 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.479301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerStarted","Data":"266b4b43936577b152809b2faeced2fab045a2d24f7fdda235a5d61c960cb3c9"}
Jul 07 19:00:49.481128 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.481074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m64kp" event={"ID":"23595584-cbfb-4ad5-985a-449f84e148c0","Type":"ContainerStarted","Data":"d6d4b0667a8f6155c35b3c731a489518114a324f2f3290059c394cafb08b1eeb"}
Jul 07 19:00:49.482728 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.482687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lp9zc" event={"ID":"6c2bd74c-a302-4fd8-94d0-c41ed8537fed","Type":"ContainerStarted","Data":"4b45642625521589849aff4d6259a7137af6bb7edd8851445f4622ff0d787619"}
Jul 07 19:00:49.485502 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.485472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n" event={"ID":"5ec87123-2d72-49e9-987c-195a580ebf84","Type":"ContainerStarted","Data":"19d8ac66d049d9b4dcb8f60985e6ad848e73e51682a65f91d78ca38af82b497a"}
Jul 07 19:00:49.487279 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.487254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n54g5" event={"ID":"b852f3ea-6aa4-4e1e-82db-92562d7b5fa4","Type":"ContainerStarted","Data":"f5ed6ed32cb5e4ab9cf6bf466c689ab097d648153a09873729fd0e1fbbf21b0b"}
Jul 07 19:00:49.489236 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.489209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxzgf" event={"ID":"49533686-cfa9-4fdb-80c5-38fd831c76ed","Type":"ContainerStarted","Data":"b7b87adf230ea3dd3f3ffb1bbbb96b88c35bb9c19598ee458e1b7fbcd0bae0ef"}
Jul 07 19:00:49.490940 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.490889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal" event={"ID":"922e395221ca99f0b2c4a3815455d230","Type":"ContainerStarted","Data":"7a24379f5d8a6278f545bc7e2c338d82fd7f81584c7775682da5aca194f6f1f2"}
Jul 07 19:00:49.519352 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.519306    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lp9zc" podStartSLOduration=3.888414003 podStartE2EDuration="14.519290011s" totalImagesPullingTime="10.630876008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:49.519264165 +0000 UTC m=+15.239063356" watchObservedRunningTime="2026-07-07 19:00:49.519290011 +0000 UTC m=+15.239089202"
Jul 07 19:00:49.534953 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.534915    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m64kp" podStartSLOduration=3.977303422 podStartE2EDuration="14.534903192s" totalImagesPullingTime="10.55759977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:49.53473544 +0000 UTC m=+15.254534624" watchObservedRunningTime="2026-07-07 19:00:49.534903192 +0000 UTC m=+15.254702382"
Jul 07 19:00:49.571073 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.571034    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cxzgf" podStartSLOduration=3.768498541 podStartE2EDuration="14.571022267s" totalImagesPullingTime="10.802523726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:49.570876572 +0000 UTC m=+15.290675767" watchObservedRunningTime="2026-07-07 19:00:49.571022267 +0000 UTC m=+15.290821456"
Jul 07 19:00:49.593200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:49.593158    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n54g5" podStartSLOduration=4.925327563 podStartE2EDuration="14.593141314s" totalImagesPullingTime="9.667813751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:49.592938516 +0000 UTC m=+15.312737781" watchObservedRunningTime="2026-07-07 19:00:49.593141314 +0000 UTC m=+15.312940505"
Jul 07 19:00:50.402489 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:50.402298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:50.402657 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:50.402369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:50.402657 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:50.402567    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:50.402811 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:50.402666    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:51.123866 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.123823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:51.403009 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.402942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:51.403185 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:51.403060    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:51.838147 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.838062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:51.838691 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.838671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:51.925324 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.925299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:51.925473 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:51.925330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:51.925473 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:51.925437    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:51.925473 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:51.925448    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:51.925598 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:51.925496    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:01:07.925480151 +0000 UTC m=+33.645279322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:00:51.925598 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:51.925509    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:07.925503713 +0000 UTC m=+33.645302882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:00:52.026418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:52.026392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:52.026541 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.026531    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:00:52.026595 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.026547    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:00:52.026595 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.026559    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:52.026691 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.026607    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:08.02659083 +0000 UTC m=+33.746390023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:00:52.402920 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:52.402893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:52.403569 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.403000    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:52.403569 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:52.403059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:52.403569 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:52.403149    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:52.499533 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:52.499500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n54g5"
Jul 07 19:00:53.402099 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:53.402025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:53.402247 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:53.402146    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:53.501883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:53.501851    2560 generic.go:356] "Generic (PLEG): container finished" podID="922e395221ca99f0b2c4a3815455d230" containerID="7a24379f5d8a6278f545bc7e2c338d82fd7f81584c7775682da5aca194f6f1f2" exitCode=0
Jul 07 19:00:53.502465 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:53.501927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal" event={"ID":"922e395221ca99f0b2c4a3815455d230","Type":"ContainerDied","Data":"7a24379f5d8a6278f545bc7e2c338d82fd7f81584c7775682da5aca194f6f1f2"}
Jul 07 19:00:54.402372 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:54.402310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:54.402525 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:54.402432    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:54.402525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:54.402485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:54.402652 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:54.402591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:55.403375 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:55.403342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:55.403800 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:55.403435    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:56.403047 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:56.403015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:56.403047 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:56.403061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:56.403252 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:56.403171    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:56.403297 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:56.403248    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:57.365905 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.365771    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 19:00:57.402928 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.402906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:57.403035 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:57.403005    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:00:57.511455 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.511428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal" event={"ID":"922e395221ca99f0b2c4a3815455d230","Type":"ContainerStarted","Data":"596cdfd5242ffe7db10343ec8d2e74bc49871241e2c2c413ac62e53a95e010b2"}
Jul 07 19:00:57.513054 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.513034    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="266b4b43936577b152809b2faeced2fab045a2d24f7fdda235a5d61c960cb3c9" exitCode=0
Jul 07 19:00:57.513128 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.513104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"266b4b43936577b152809b2faeced2fab045a2d24f7fdda235a5d61c960cb3c9"}
Jul 07 19:00:57.517855 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.517833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"6ace01e5a5b1691e5f622f9bcdd438292e23e63e81c0836970f8cf91e9a2cce5"}
Jul 07 19:00:57.517953 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.517906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"d1aacfe039062b0cb2c990774d95cfcf0dcfdb18f86d770fff8573cddea818f2"}
Jul 07 19:00:57.518020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.517961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"15bb409a54038529095720079199220f890864cdbd8af9f397b07ca67df9f615"}
Jul 07 19:00:57.518020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.518008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"8ef00f8a62d66722697af31e1fe12b0fb44f7783bec9ca4633e0299cd421e571"}
Jul 07 19:00:57.519751 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.519727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n" event={"ID":"5ec87123-2d72-49e9-987c-195a580ebf84","Type":"ContainerStarted","Data":"0270ed045ece1ef78b7150d95db32eaef88d2439db8c6ed713cde6b6eda55437"}
Jul 07 19:00:57.521460 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.521438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t65ms" event={"ID":"b75be16b-9497-4bfe-8e37-9dbffc5ce222","Type":"ContainerStarted","Data":"1b178d6f521cd80cb228c6a9cec25951da3d6e63582e38217d3e8633c940a656"}
Jul 07 19:00:57.528552 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.528448    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-56.ec2.internal" podStartSLOduration=22.528436705 podStartE2EDuration="22.528436705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:57.527922048 +0000 UTC m=+23.247721401" watchObservedRunningTime="2026-07-07 19:00:57.528436705 +0000 UTC m=+23.248235892"
Jul 07 19:00:57.591881 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:57.591845    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t65ms" podStartSLOduration=1.945786468 podStartE2EDuration="22.591824369s" totalImagesPullingTime="20.646037901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:57.591747148 +0000 UTC m=+23.311546335" watchObservedRunningTime="2026-07-07 19:00:57.591824369 +0000 UTC m=+23.311623560"
Jul 07 19:00:58.287645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.287377    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:00:57.365891858Z","UUID":"82f9770b-4aea-489d-940f-c13a4abb04cd","Handler":null,"Name":"","Endpoint":""}
Jul 07 19:00:58.288942 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.288921    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 19:00:58.289080 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.288974    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 19:00:58.402413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.402387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:00:58.402926 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:58.402501    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:00:58.402926 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.402568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:00:58.402926 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:58.402682    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:00:58.526619 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.526584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"e224da2622fad2da23820ffcd140b9832b9ce708169f3a94cb4446794f663e96"}
Jul 07 19:00:58.526787 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:58.526670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"b3f2921ddbec87c4136ac7b710310f746fd4ff6f8121cc6cfe22c48e5f04e7e3"}
Jul 07 19:00:59.403023 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:00:59.402996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:00:59.403470 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:00:59.403115    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:00.402948 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:00.402876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:00.403087 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:00.402998    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:00.403087 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:00.403060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:00.403537 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:00.403167    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:00.536439 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:00.536384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"7638f3260240f0a893fc74ed3e60eb9b65ddfc9a46cee29af1ab7a64bcc59acf"}
Jul 07 19:01:00.538594 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:00.538557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n" event={"ID":"5ec87123-2d72-49e9-987c-195a580ebf84","Type":"ContainerStarted","Data":"9a300deb0271e4a06d202a9dadb9b6804c94b20f56889aae5c3a1fde33d1231e"}
Jul 07 19:01:00.558264 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:00.558217    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h565n" podStartSLOduration=3.7228333129999998 podStartE2EDuration="25.558204209s" totalImagesPullingTime="21.835370896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:00.557847411 +0000 UTC m=+26.277646603" watchObservedRunningTime="2026-07-07 19:01:00.558204209 +0000 UTC m=+26.278003399"
Jul 07 19:01:01.402442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:01.402409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:01.402613 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:01.402545    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:02.402797 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.402594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:02.403437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.402646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:02.403437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:02.402917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:02.403437 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:02.402942    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:02.544443 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.544365    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="f2e972d9047430438ee218b4a9432927759387daab3cd8cf0047ccdb225ca6fa" exitCode=0
Jul 07 19:01:02.544608 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.544433    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"f2e972d9047430438ee218b4a9432927759387daab3cd8cf0047ccdb225ca6fa"}
Jul 07 19:01:02.548104 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.548081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" event={"ID":"9e985872-3005-4741-95a2-c57a7335b8bc","Type":"ContainerStarted","Data":"bbf30ca1840ed6933f6195df926a6e0a0124f4f5470d8298c04b922617b3d392"}
Jul 07 19:01:02.548325 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.548304    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:02.548491 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.548356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:02.562718 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.562686    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:02.601811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:02.601777    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf" podStartSLOduration=7.194491802 podStartE2EDuration="27.601765452s" totalImagesPullingTime="20.40727365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:02.600229411 +0000 UTC m=+28.320028601" watchObservedRunningTime="2026-07-07 19:01:02.601765452 +0000 UTC m=+28.321564635"
Jul 07 19:01:03.402370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:03.402343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:03.402470 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:03.402450    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:03.552853 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:03.552783    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="a1cd7866ddf781679f0c704ef919a908104507193f28580b97a1ea9cc55e970e" exitCode=0
Jul 07 19:01:03.553159 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:03.552860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"a1cd7866ddf781679f0c704ef919a908104507193f28580b97a1ea9cc55e970e"}
Jul 07 19:01:03.553262 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:03.553244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:03.567294 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:03.567273    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:04.009763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.009731    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-klwmk"]
Jul 07 19:01:04.009938 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.009846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:04.010012 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:04.009972    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:04.012715 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.012673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xdrgp"]
Jul 07 19:01:04.012833 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.012802    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:04.012940 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:04.012916    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:04.013381 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.013360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pskjn"]
Jul 07 19:01:04.013530 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.013455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:04.013605 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:04.013541    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:04.559666 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.559588    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="a65c69656808d1d5bbd2776bd7fcdaadaf6540bb6de9f50cf8ed681b3bf41969" exitCode=0
Jul 07 19:01:04.559666 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:04.559653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"a65c69656808d1d5bbd2776bd7fcdaadaf6540bb6de9f50cf8ed681b3bf41969"}
Jul 07 19:01:05.403579 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:05.403319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:05.403770 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:05.403408    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:05.403770 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:05.403653    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:05.403770 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:05.403459    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:05.403770 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:05.403759    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:05.403986 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:05.403791    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:07.402822 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:07.402789    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:07.402822 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:07.402823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:07.403441 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:07.402790    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:07.403441 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.402900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:07.403441 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.402978    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:07.403441 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.403088    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:07.953413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:07.953377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:07.953593 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:07.953428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:07.953593 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.953519    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:01:07.953593 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.953556    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:01:07.953593 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.953587    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret podName:443ebe9b-7063-4620-9025-3884c0734c8c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:39.953569853 +0000 UTC m=+65.673369062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret") pod "global-pull-secret-syncer-klwmk" (UID: "443ebe9b-7063-4620-9025-3884c0734c8c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:01:07.953768 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:07.953604    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:01:39.953596949 +0000 UTC m=+65.673396124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:01:08.054123 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:08.054087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:08.054290 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:08.054270    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:01:08.054353 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:08.054301    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:01:08.054353 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:08.054314    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n2n9c for pod openshift-network-diagnostics/network-check-target-xdrgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:01:08.054450 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:08.054372    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c podName:e797c505-ee21-4883-bf3c-0cf692c2dd95 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:40.05435374 +0000 UTC m=+65.774152933 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n2n9c" (UniqueName: "kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c") pod "network-check-target-xdrgp" (UID: "e797c505-ee21-4883-bf3c-0cf692c2dd95") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:01:09.401978 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.401949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:09.402544 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:09.402061    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klwmk" podUID="443ebe9b-7063-4620-9025-3884c0734c8c"
Jul 07 19:01:09.402544 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.402104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:09.402544 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:09.402205    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pskjn" podUID="5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed"
Jul 07 19:01:09.402544 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.402230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:09.402544 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:09.402316    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdrgp" podUID="e797c505-ee21-4883-bf3c-0cf692c2dd95"
Jul 07 19:01:09.573653 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.573605    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="a9a5f122330602f56ce22f429338a84cf84d1fe39e59359f609039c17253b71b" exitCode=0
Jul 07 19:01:09.573746 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.573656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"a9a5f122330602f56ce22f429338a84cf84d1fe39e59359f609039c17253b71b"}
Jul 07 19:01:09.843156 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.843106    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-56.ec2.internal" event="NodeReady"
Jul 07 19:01:09.843249 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.843213    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 19:01:09.896355 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.896333    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zrz2k"]
Jul 07 19:01:09.908361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.908341    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5l8j2"]
Jul 07 19:01:09.908494 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.908477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:09.912177 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.912158    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\""
Jul 07 19:01:09.912389 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.912371    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 19:01:09.912772 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.912753    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 19:01:09.920851 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.920834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrz2k"]
Jul 07 19:01:09.920937 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.920864    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5l8j2"]
Jul 07 19:01:09.920937 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.920921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:09.922809 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.922792    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 19:01:09.922809 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.922801    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 19:01:09.922934 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.922874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hwc8x\""
Jul 07 19:01:09.923217 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:09.923204    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 19:01:10.067778 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.067756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m875k\" (UniqueName: \"kubernetes.io/projected/7a897e22-b61b-4495-b468-047d2f3bc6b5-kube-api-access-m875k\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.067883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.067781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-tmp-dir\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.067883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.067816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-config-volume\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.068003 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.067916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gfj\" (UniqueName: \"kubernetes.io/projected/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-kube-api-access-w9gfj\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.068003 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.067952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.068090 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.068008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a897e22-b61b-4495-b468-047d2f3bc6b5-cert\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.168617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-tmp-dir\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.168617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-config-volume\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.168789 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gfj\" (UniqueName: \"kubernetes.io/projected/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-kube-api-access-w9gfj\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.168789 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.168789 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a897e22-b61b-4495-b468-047d2f3bc6b5-cert\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.168958 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m875k\" (UniqueName: \"kubernetes.io/projected/7a897e22-b61b-4495-b468-047d2f3bc6b5-kube-api-access-m875k\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.168958 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:10.168880    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:10.168958 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:10.168940    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:10.668923283 +0000 UTC m=+36.388722470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:10.168958 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.168880    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-tmp-dir\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.169203 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.169182    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-config-volume\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.172866 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.172848    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a897e22-b61b-4495-b468-047d2f3bc6b5-cert\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.186035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.186018    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gfj\" (UniqueName: \"kubernetes.io/projected/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-kube-api-access-w9gfj\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.186396 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.186380    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m875k\" (UniqueName: \"kubernetes.io/projected/7a897e22-b61b-4495-b468-047d2f3bc6b5-kube-api-access-m875k\") pod \"ingress-canary-5l8j2\" (UID: \"7a897e22-b61b-4495-b468-047d2f3bc6b5\") " pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.228426 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.228410    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5l8j2"
Jul 07 19:01:10.390974 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.390821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5l8j2"]
Jul 07 19:01:10.394247 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:10.394218    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a897e22_b61b_4495_b468_047d2f3bc6b5.slice/crio-1ede207ed72062778e5dd9ccb09e3f871643b90af49417d24db44f771a12d366 WatchSource:0}: Error finding container 1ede207ed72062778e5dd9ccb09e3f871643b90af49417d24db44f771a12d366: Status 404 returned error can't find the container with id 1ede207ed72062778e5dd9ccb09e3f871643b90af49417d24db44f771a12d366
Jul 07 19:01:10.577549 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.577512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5l8j2" event={"ID":"7a897e22-b61b-4495-b468-047d2f3bc6b5","Type":"ContainerStarted","Data":"1ede207ed72062778e5dd9ccb09e3f871643b90af49417d24db44f771a12d366"}
Jul 07 19:01:10.580508 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.580488    2560 generic.go:356] "Generic (PLEG): container finished" podID="dddb970a-248c-4fb2-aca6-49698d3f75da" containerID="4e211171a9a23fbdca0de3d44b19011ac58daac3cafda73431af81a146b8dd44" exitCode=0
Jul 07 19:01:10.580596 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.580515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerDied","Data":"4e211171a9a23fbdca0de3d44b19011ac58daac3cafda73431af81a146b8dd44"}
Jul 07 19:01:10.672351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:10.672326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:10.672519 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:10.672499    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:10.672591 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:10.672576    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:11.672552209 +0000 UTC m=+37.392351400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:11.402256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.402222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:11.402436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.402222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:11.402436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.402222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:11.405169 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.405142    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 19:01:11.405785 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.405767    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 19:01:11.405785 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.405777    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\""
Jul 07 19:01:11.405948 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.405780    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 19:01:11.405948 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.405857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\""
Jul 07 19:01:11.406165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.406147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 19:01:11.587412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.587385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" event={"ID":"dddb970a-248c-4fb2-aca6-49698d3f75da","Type":"ContainerStarted","Data":"81f960b40733c4a518a1cd59ae5accbffa525e07a7ed2741367ab2b3da0039d6"}
Jul 07 19:01:11.618444 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.618398    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xxnjb" podStartSLOduration=15.266619538 podStartE2EDuration="36.618380615s" totalImagesPullingTime="21.351761077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:11.616442734 +0000 UTC m=+37.336241924" watchObservedRunningTime="2026-07-07 19:01:11.618380615 +0000 UTC m=+37.338179808"
Jul 07 19:01:11.679630 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:11.679556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:11.679823 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:11.679800    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:11.679946 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:11.679879    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:13.679857852 +0000 UTC m=+39.399657029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:13.593099 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:13.593068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5l8j2" event={"ID":"7a897e22-b61b-4495-b468-047d2f3bc6b5","Type":"ContainerStarted","Data":"4d0b17d0aefd34b989805daecdefe9932576bc17754a89d3ed4921a21e8ff4cb"}
Jul 07 19:01:13.612132 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:13.612089    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5l8j2" podStartSLOduration=1.840915941 podStartE2EDuration="4.61207361s" totalImagesPullingTime="2.771157669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:13.610756335 +0000 UTC m=+39.330555526" watchObservedRunningTime="2026-07-07 19:01:13.61207361 +0000 UTC m=+39.331872800"
Jul 07 19:01:13.695056 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:13.695027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:13.695175 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:13.695157    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:13.695223 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:13.695216    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:17.695200113 +0000 UTC m=+43.414999281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:17.722058 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:17.722019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:17.722457 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:17.722144    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:17.722457 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:17.722209    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:25.722194404 +0000 UTC m=+51.441993571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:25.498294 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.498266    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"]
Jul 07 19:01:25.509374 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.509341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.510272 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.510251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"]
Jul 07 19:01:25.511616 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.511600    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jv7xq\""
Jul 07 19:01:25.511851 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.511824    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 19:01:25.511934 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.511872    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 19:01:25.512386 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.512368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:01:25.512632 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.512613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 19:01:25.605629 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.605601    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64bcb7459f-kdll9"]
Jul 07 19:01:25.620297 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.620275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.623027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.623000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64bcb7459f-kdll9"]
Jul 07 19:01:25.624737 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.624693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9ccvr\""
Jul 07 19:01:25.624848 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.624817    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 19:01:25.625082 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.625059    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 07 19:01:25.625344 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.625327    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 07 19:01:25.625588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.625574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 07 19:01:25.625669 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.625594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 07 19:01:25.625669 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.625638    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 19:01:25.668664 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.668643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/214887a9-0643-419b-80d2-ffd422948adc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.668781 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.668671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswr9\" (UniqueName: \"kubernetes.io/projected/214887a9-0643-419b-80d2-ffd422948adc-kube-api-access-lswr9\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.668781 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.668730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/214887a9-0643-419b-80d2-ffd422948adc-config\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.769999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.769955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-stats-auth\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.769999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.769992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/214887a9-0643-419b-80d2-ffd422948adc-config\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4lr\" (UniqueName: \"kubernetes.io/projected/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-kube-api-access-ff4lr\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/214887a9-0643-419b-80d2-ffd422948adc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.770136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lswr9\" (UniqueName: \"kubernetes.io/projected/214887a9-0643-419b-80d2-ffd422948adc-kube-api-access-lswr9\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.770431 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.770145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-default-certificate\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.770431 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:25.770182    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:25.770431 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:25.770324    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.770304652 +0000 UTC m=+67.490103824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:25.774034 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.774017    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/214887a9-0643-419b-80d2-ffd422948adc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.779848 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.779822    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/214887a9-0643-419b-80d2-ffd422948adc-config\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.786877 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.786858    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswr9\" (UniqueName: \"kubernetes.io/projected/214887a9-0643-419b-80d2-ffd422948adc-kube-api-access-lswr9\") pod \"service-ca-operator-5d46d9ffd4-f9qgd\" (UID: \"214887a9-0643-419b-80d2-ffd422948adc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.819975 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.819958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"
Jul 07 19:01:25.870910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.870875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-default-certificate\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.871001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.870940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-stats-auth\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.871001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.870989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.871079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.871026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4lr\" (UniqueName: \"kubernetes.io/projected/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-kube-api-access-ff4lr\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.871134 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:25.871108    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:25.871199 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:25.871186    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:26.371153221 +0000 UTC m=+52.090952388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:25.871262 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.871214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.871362 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:25.871348    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:26.371330752 +0000 UTC m=+52.091129922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:25.873252 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.873233    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-default-certificate\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.873336 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.873309    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-stats-auth\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.880663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.880640    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4lr\" (UniqueName: \"kubernetes.io/projected/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-kube-api-access-ff4lr\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:25.963512 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:25.963485    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd"]
Jul 07 19:01:25.966731 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:25.966696    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214887a9_0643_419b_80d2_ffd422948adc.slice/crio-0390b8859ac037a058116da9684dc2380599562d2b8acd7342ff2c0486366846 WatchSource:0}: Error finding container 0390b8859ac037a058116da9684dc2380599562d2b8acd7342ff2c0486366846: Status 404 returned error can't find the container with id 0390b8859ac037a058116da9684dc2380599562d2b8acd7342ff2c0486366846
Jul 07 19:01:26.375322 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:26.375296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:26.375456 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:26.375350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:26.375494 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:26.375464    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:26.375529 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:26.375464    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:27.37544455 +0000 UTC m=+53.095243737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:26.375529 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:26.375513    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:27.375502151 +0000 UTC m=+53.095301319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:26.625615 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:26.625543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd" event={"ID":"214887a9-0643-419b-80d2-ffd422948adc","Type":"ContainerStarted","Data":"0390b8859ac037a058116da9684dc2380599562d2b8acd7342ff2c0486366846"}
Jul 07 19:01:27.194921 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.194892    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"]
Jul 07 19:01:27.210410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.210374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"]
Jul 07 19:01:27.210531 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.210466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"
Jul 07 19:01:27.214772 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.214750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-g4njk\""
Jul 07 19:01:27.382470 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.382443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:27.382593 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.382495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v5jz\" (UniqueName: \"kubernetes.io/projected/1a77a3f9-415b-42f9-86a0-5102581762c4-kube-api-access-9v5jz\") pod \"network-check-source-989f8b8dc-rr5zx\" (UID: \"1a77a3f9-415b-42f9-86a0-5102581762c4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"
Jul 07 19:01:27.382593 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.382516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:27.382725 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:27.382596    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:27.382725 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:27.382616    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:29.382598104 +0000 UTC m=+55.102397277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:27.382725 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:27.382637    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:29.382630698 +0000 UTC m=+55.102429866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:27.483520 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.483421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v5jz\" (UniqueName: \"kubernetes.io/projected/1a77a3f9-415b-42f9-86a0-5102581762c4-kube-api-access-9v5jz\") pod \"network-check-source-989f8b8dc-rr5zx\" (UID: \"1a77a3f9-415b-42f9-86a0-5102581762c4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"
Jul 07 19:01:27.494390 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.494372    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v5jz\" (UniqueName: \"kubernetes.io/projected/1a77a3f9-415b-42f9-86a0-5102581762c4-kube-api-access-9v5jz\") pod \"network-check-source-989f8b8dc-rr5zx\" (UID: \"1a77a3f9-415b-42f9-86a0-5102581762c4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"
Jul 07 19:01:27.519141 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.519119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"
Jul 07 19:01:27.672689 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:27.672664    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx"]
Jul 07 19:01:27.675980 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:27.675933    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a77a3f9_415b_42f9_86a0_5102581762c4.slice/crio-545e6658440da548a1ce963650cbdc0cc55e8a598d2221a7657feec3c30908cf WatchSource:0}: Error finding container 545e6658440da548a1ce963650cbdc0cc55e8a598d2221a7657feec3c30908cf: Status 404 returned error can't find the container with id 545e6658440da548a1ce963650cbdc0cc55e8a598d2221a7657feec3c30908cf
Jul 07 19:01:28.633274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:28.633235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx" event={"ID":"1a77a3f9-415b-42f9-86a0-5102581762c4","Type":"ContainerStarted","Data":"545e6658440da548a1ce963650cbdc0cc55e8a598d2221a7657feec3c30908cf"}
Jul 07 19:01:29.398859 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.398821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:29.399293 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.398919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:29.399293 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.399006    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.398979805 +0000 UTC m=+59.118778993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:29.399293 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.399032    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:29.399293 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.399083    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.39906538 +0000 UTC m=+59.118864572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:29.638047 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.638010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd" event={"ID":"214887a9-0643-419b-80d2-ffd422948adc","Type":"ContainerStarted","Data":"79ccb1d7e86bbc99e168726c17e82545c07f7a664c645644add6f80b4e5f2b25"}
Jul 07 19:01:29.691857 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.691768    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qrdxq"]
Jul 07 19:01:29.694051 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.694024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.700328 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.700298    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-bj6vq\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\"" type="*v1.Secret"
Jul 07 19:01:29.700960 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.700923    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 07 19:01:29.701066 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.701043    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 07 19:01:29.701256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.701238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbtb\" (UniqueName: \"kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.701321 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.701304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.701391 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.701360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97fde49a-d038-462a-bcee-cc86d2cec0ec-crio-socket\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.701471 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.701452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97fde49a-d038-462a-bcee-cc86d2cec0ec-data-volume\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.701517 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.701502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.701592 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.701566    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Jul 07 19:01:29.702214 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:29.702189    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 07 19:01:29.763391 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.763330    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qrdxq"]
Jul 07 19:01:29.763947 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.763899    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd" podStartSLOduration=1.8474888219999999 podStartE2EDuration="4.763879626s" totalImagesPullingTime="2.916390804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:29.759513937 +0000 UTC m=+55.479313126" watchObservedRunningTime="2026-07-07 19:01:29.763879626 +0000 UTC m=+55.483678816"
Jul 07 19:01:29.802335 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbtb\" (UniqueName: \"kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802539 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802539 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97fde49a-d038-462a-bcee-cc86d2cec0ec-crio-socket\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802539 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97fde49a-d038-462a-bcee-cc86d2cec0ec-data-volume\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802566    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97fde49a-d038-462a-bcee-cc86d2cec0ec-crio-socket\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:29.802833 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:29.802814    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97fde49a-d038-462a-bcee-cc86d2cec0ec-data-volume\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:30.705579 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.705546    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 19:01:30.802884 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.802636    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition
Jul 07 19:01:30.802884 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.802731    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.30269306 +0000 UTC m=+57.022492227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : failed to sync configmap cache: timed out waiting for the condition
Jul 07 19:01:30.802884 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.802756    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Jul 07 19:01:30.802884 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.802829    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.302811371 +0000 UTC m=+57.022610545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : failed to sync secret cache: timed out waiting for the condition
Jul 07 19:01:30.813223 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.813202    2560 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 07 19:01:30.813278 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.813238    2560 projected.go:196] Error preparing data for projected volume kube-api-access-dlbtb for pod openshift-insights/insights-runtime-extractor-qrdxq: failed to sync configmap cache: timed out waiting for the condition
Jul 07 19:01:30.813312 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:30.813289    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.313274378 +0000 UTC m=+57.033073563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dlbtb" (UniqueName: "kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : failed to sync configmap cache: timed out waiting for the condition
Jul 07 19:01:30.872952 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.872921    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"]
Jul 07 19:01:30.875001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.874982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"
Jul 07 19:01:30.877388 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.877365    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8j66g\""
Jul 07 19:01:30.877487 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.877426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 19:01:30.877814 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.877799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 19:01:30.887287 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.887267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"]
Jul 07 19:01:30.909836 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:30.909815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5hx\" (UniqueName: \"kubernetes.io/projected/375315b8-a70a-4338-8890-f099338e623d-kube-api-access-zs5hx\") pod \"migrator-d57d58cc9-29qsq\" (UID: \"375315b8-a70a-4338-8890-f099338e623d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"
Jul 07 19:01:31.007005 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.006984    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 19:01:31.010749 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.010730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5hx\" (UniqueName: \"kubernetes.io/projected/375315b8-a70a-4338-8890-f099338e623d-kube-api-access-zs5hx\") pod \"migrator-d57d58cc9-29qsq\" (UID: \"375315b8-a70a-4338-8890-f099338e623d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"
Jul 07 19:01:31.019197 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.019172    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5hx\" (UniqueName: \"kubernetes.io/projected/375315b8-a70a-4338-8890-f099338e623d-kube-api-access-zs5hx\") pod \"migrator-d57d58cc9-29qsq\" (UID: \"375315b8-a70a-4338-8890-f099338e623d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"
Jul 07 19:01:31.090133 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.090103    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\""
Jul 07 19:01:31.110879 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.110851    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 19:01:31.183777 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.183752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"
Jul 07 19:01:31.192652 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.192625    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 19:01:31.305356 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.305316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq"]
Jul 07 19:01:31.309433 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:31.309399    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375315b8_a70a_4338_8890_f099338e623d.slice/crio-96f9ed34c8e2f618f4e9ed379627011079b28f0e0b206fba45842e6d13452ec2 WatchSource:0}: Error finding container 96f9ed34c8e2f618f4e9ed379627011079b28f0e0b206fba45842e6d13452ec2: Status 404 returned error can't find the container with id 96f9ed34c8e2f618f4e9ed379627011079b28f0e0b206fba45842e6d13452ec2
Jul 07 19:01:31.312465 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.312413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:31.312575 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.312523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:31.312656 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:31.312638    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:31.312656 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:31.312651    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:01:31.312788 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:31.312730    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:32.31268447 +0000 UTC m=+58.032483661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:31.312788 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:31.312754    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:32.312742478 +0000 UTC m=+58.032541648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : configmap "kube-rbac-proxy" not found
Jul 07 19:01:31.413200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.413174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbtb\" (UniqueName: \"kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:31.415441 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.415423    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbtb\" (UniqueName: \"kubernetes.io/projected/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-api-access-dlbtb\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:31.447571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.447549    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lp9zc_6c2bd74c-a302-4fd8-94d0-c41ed8537fed/dns-node-resolver/0.log"
Jul 07 19:01:31.644274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.644173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx" event={"ID":"1a77a3f9-415b-42f9-86a0-5102581762c4","Type":"ContainerStarted","Data":"8e2ce6882be9eafd51c7226ed349ca44383554ff5d18b6d87be2a5318b336540"}
Jul 07 19:01:31.645298 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.645273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq" event={"ID":"375315b8-a70a-4338-8890-f099338e623d","Type":"ContainerStarted","Data":"96f9ed34c8e2f618f4e9ed379627011079b28f0e0b206fba45842e6d13452ec2"}
Jul 07 19:01:31.663325 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:31.663279    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-rr5zx" podStartSLOduration=1.537389873 podStartE2EDuration="4.663265068s" totalImagesPullingTime="3.125875195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:31.662304335 +0000 UTC m=+57.382103525" watchObservedRunningTime="2026-07-07 19:01:31.663265068 +0000 UTC m=+57.383064298"
Jul 07 19:01:32.320382 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:32.320341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:32.320771 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:32.320428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:32.320771 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:32.320464    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:01:32.320771 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:32.320542    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:34.32052553 +0000 UTC m=+60.040324706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : configmap "kube-rbac-proxy" not found
Jul 07 19:01:32.320771 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:32.320580    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:32.320771 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:32.320636    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:34.320624224 +0000 UTC m=+60.040423409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:32.446075 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:32.446040    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m64kp_23595584-cbfb-4ad5-985a-449f84e148c0/node-ca/0.log"
Jul 07 19:01:33.428346 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.428323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:33.428618 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.428404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:33.428618 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:33.428475    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.428457113 +0000 UTC m=+67.148256289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:33.428618 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:33.428526    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:33.428618 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:33.428568    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.428557602 +0000 UTC m=+67.148356773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:33.447827 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.447802    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5l8j2_7a897e22-b61b-4495-b468-047d2f3bc6b5/serve-healthcheck-canary/0.log"
Jul 07 19:01:33.652573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.652509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq" event={"ID":"375315b8-a70a-4338-8890-f099338e623d","Type":"ContainerStarted","Data":"7f7354c5083a93eee408aa702f616aa13cf93ab9ddc451052038d8233b919a73"}
Jul 07 19:01:33.652573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.652558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq" event={"ID":"375315b8-a70a-4338-8890-f099338e623d","Type":"ContainerStarted","Data":"179ab272cd25ae8b5a1adddefce7f59898a29031f488fe56714abeb401be1bf3"}
Jul 07 19:01:33.682062 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:33.682017    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-29qsq" podStartSLOduration=1.627516411 podStartE2EDuration="3.682002756s" totalImagesPullingTime="2.054486345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:33.680778758 +0000 UTC m=+59.400577939" watchObservedRunningTime="2026-07-07 19:01:33.682002756 +0000 UTC m=+59.401801946"
Jul 07 19:01:34.334717 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:34.334676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:34.334876 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:34.334786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:34.334876 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:34.334821    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:34.334946 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:34.334884    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:01:34.334946 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:34.334895    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:38.334879191 +0000 UTC m=+64.054678359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:34.334946 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:34.334924    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:38.334913116 +0000 UTC m=+64.054712287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : configmap "kube-rbac-proxy" not found
Jul 07 19:01:35.572054 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:35.572025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28xf"
Jul 07 19:01:38.368309 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:38.368273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:38.368762 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:38.368352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:38.368762 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:38.368435    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:38.368762 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:38.368457    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:01:38.368762 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:38.368517    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:46.368497687 +0000 UTC m=+72.088296875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:38.368762 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:38.368540    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:01:46.368529202 +0000 UTC m=+72.088328401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : configmap "kube-rbac-proxy" not found
Jul 07 19:01:39.980573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:39.980535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:39.980573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:39.980571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:01:39.987804 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:39.987782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 19:01:39.990818 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:39.990797    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 19:01:39.990888 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:39.990860    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs podName:5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed nodeName:}" failed. No retries permitted until 2026-07-07 19:02:43.990841682 +0000 UTC m=+129.710640849 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs") pod "network-metrics-daemon-pskjn" (UID: "5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed") : secret "metrics-daemon-secret" not found
Jul 07 19:01:39.990888 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:39.990857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 19:01:40.003046 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.003027    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/443ebe9b-7063-4620-9025-3884c0734c8c-original-pull-secret\") pod \"global-pull-secret-syncer-klwmk\" (UID: \"443ebe9b-7063-4620-9025-3884c0734c8c\") " pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:40.081607 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.081571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:40.083908 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.083882    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2n9c\" (UniqueName: \"kubernetes.io/projected/e797c505-ee21-4883-bf3c-0cf692c2dd95-kube-api-access-n2n9c\") pod \"network-check-target-xdrgp\" (UID: \"e797c505-ee21-4883-bf3c-0cf692c2dd95\") " pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:40.213347 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.213314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klwmk"
Jul 07 19:01:40.233180 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.233119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\""
Jul 07 19:01:40.240637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.240619    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:40.347196 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.347132    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-klwmk"]
Jul 07 19:01:40.352077 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:40.352052    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443ebe9b_7063_4620_9025_3884c0734c8c.slice/crio-1f9d4b043f7b85507a547694f5074789d898ec4819534c54beae0fad9a28c5b7 WatchSource:0}: Error finding container 1f9d4b043f7b85507a547694f5074789d898ec4819534c54beae0fad9a28c5b7: Status 404 returned error can't find the container with id 1f9d4b043f7b85507a547694f5074789d898ec4819534c54beae0fad9a28c5b7
Jul 07 19:01:40.397589 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.397539    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xdrgp"]
Jul 07 19:01:40.399397 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:40.399378    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode797c505_ee21_4883_bf3c_0cf692c2dd95.slice/crio-bf78fbdabe167ee7240b0b38b73d61c07ead7f23d9dea62a399b6645e5cc50fe WatchSource:0}: Error finding container bf78fbdabe167ee7240b0b38b73d61c07ead7f23d9dea62a399b6645e5cc50fe: Status 404 returned error can't find the container with id bf78fbdabe167ee7240b0b38b73d61c07ead7f23d9dea62a399b6645e5cc50fe
Jul 07 19:01:40.670478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.670446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-klwmk" event={"ID":"443ebe9b-7063-4620-9025-3884c0734c8c","Type":"ContainerStarted","Data":"1f9d4b043f7b85507a547694f5074789d898ec4819534c54beae0fad9a28c5b7"}
Jul 07 19:01:40.671808 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.671780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xdrgp" event={"ID":"e797c505-ee21-4883-bf3c-0cf692c2dd95","Type":"ContainerStarted","Data":"c9fedd66aeaa2b5a266eaad72de77271e863eff4221976a76ef3726185cf3173"}
Jul 07 19:01:40.671937 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.671819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xdrgp" event={"ID":"e797c505-ee21-4883-bf3c-0cf692c2dd95","Type":"ContainerStarted","Data":"bf78fbdabe167ee7240b0b38b73d61c07ead7f23d9dea62a399b6645e5cc50fe"}
Jul 07 19:01:40.671937 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.671851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:01:40.729127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:40.729077    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xdrgp" podStartSLOduration=65.729062122 podStartE2EDuration="1m5.729062122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:40.728482972 +0000 UTC m=+66.448282162" watchObservedRunningTime="2026-07-07 19:01:40.729062122 +0000 UTC m=+66.448861290"
Jul 07 19:01:41.493711 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:41.493662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:41.494109 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:41.493826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:41.494109 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:41.493856    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:57.493833326 +0000 UTC m=+83.213632508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : configmap references non-existent config key: service-ca.crt
Jul 07 19:01:41.494109 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:41.493910    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 19:01:41.494109 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:41.493957    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs podName:3cd34889-f57e-46ec-a1aa-6297a9d6c8e5 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:57.493943058 +0000 UTC m=+83.213742252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs") pod "router-default-64bcb7459f-kdll9" (UID: "3cd34889-f57e-46ec-a1aa-6297a9d6c8e5") : secret "router-metrics-certs-default" not found
Jul 07 19:01:41.796254 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:41.796176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:01:41.796402 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:41.796287    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:01:41.796402 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:41.796387    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls podName:a4393ec0-a42e-4579-a6a6-6bc959d3af3c nodeName:}" failed. No retries permitted until 2026-07-07 19:02:13.796362419 +0000 UTC m=+99.516161604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls") pod "dns-default-zrz2k" (UID: "a4393ec0-a42e-4579-a6a6-6bc959d3af3c") : secret "dns-default-metrics-tls" not found
Jul 07 19:01:44.684892 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:44.684809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-klwmk" event={"ID":"443ebe9b-7063-4620-9025-3884c0734c8c","Type":"ContainerStarted","Data":"f31131a90224d3a725a4b680a2ad76805043dc3488d368f1543a04ae1fbd323e"}
Jul 07 19:01:44.702162 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:44.702118    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-klwmk" podStartSLOduration=65.706048273 podStartE2EDuration="1m9.702103891s" totalImagesPullingTime="3.996055618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:44.701678212 +0000 UTC m=+70.421477402" watchObservedRunningTime="2026-07-07 19:01:44.702103891 +0000 UTC m=+70.421903078"
Jul 07 19:01:46.433207 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:46.433172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:46.433835 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:46.433231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:46.433835 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:46.433339    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:46.433835 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:01:46.433412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls podName:97fde49a-d038-462a-bcee-cc86d2cec0ec nodeName:}" failed. No retries permitted until 2026-07-07 19:02:02.433395958 +0000 UTC m=+88.153195146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qrdxq" (UID: "97fde49a-d038-462a-bcee-cc86d2cec0ec") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:01:46.434011 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:46.433896    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97fde49a-d038-462a-bcee-cc86d2cec0ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:01:54.486173 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.486139    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"]
Jul 07 19:01:54.489196 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.489173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.494029 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.494002    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 19:01:54.496403 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.496374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 19:01:54.496582 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.496555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 19:01:54.496635 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.496601    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 19:01:54.496689 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.496656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 19:01:54.497044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.497020    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 19:01:54.503391 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.503352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 19:01:54.513380 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.513360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"]
Jul 07 19:01:54.583869 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.583845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcxg\" (UniqueName: \"kubernetes.io/projected/a0647d78-e42e-4090-9592-9f551dbc8a4d-kube-api-access-nfcxg\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.583973 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.583887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0647d78-e42e-4090-9592-9f551dbc8a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.583973 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.583931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.583973 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.583954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.584099 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.583989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.584099 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.584087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.684733 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.684696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0647d78-e42e-4090-9592-9f551dbc8a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.684840 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.684750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.684840 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.684769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.684840 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.684798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.685048 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.685026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.685107 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.685075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcxg\" (UniqueName: \"kubernetes.io/projected/a0647d78-e42e-4090-9592-9f551dbc8a4d-kube-api-access-nfcxg\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.685505 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.685469    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0647d78-e42e-4090-9592-9f551dbc8a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.687218 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.687192    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-ca\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.687388 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.687352    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.687482 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.687460    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.687540 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.687490    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0647d78-e42e-4090-9592-9f551dbc8a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.702193 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.702173    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcxg\" (UniqueName: \"kubernetes.io/projected/a0647d78-e42e-4090-9592-9f551dbc8a4d-kube-api-access-nfcxg\") pod \"cluster-proxy-proxy-agent-75778c7ccd-c2689\" (UID: \"a0647d78-e42e-4090-9592-9f551dbc8a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.808426 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.808404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"
Jul 07 19:01:54.943218 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:54.943186    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689"]
Jul 07 19:01:54.946503 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:54.946480    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0647d78_e42e_4090_9592_9f551dbc8a4d.slice/crio-a185ab6951fe5391eb261f5c791cabff83bf889ab3c5016df6cf5219f4ef6c7c WatchSource:0}: Error finding container a185ab6951fe5391eb261f5c791cabff83bf889ab3c5016df6cf5219f4ef6c7c: Status 404 returned error can't find the container with id a185ab6951fe5391eb261f5c791cabff83bf889ab3c5016df6cf5219f4ef6c7c
Jul 07 19:01:55.718274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:55.718239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689" event={"ID":"a0647d78-e42e-4090-9592-9f551dbc8a4d","Type":"ContainerStarted","Data":"a185ab6951fe5391eb261f5c791cabff83bf889ab3c5016df6cf5219f4ef6c7c"}
Jul 07 19:01:57.505013 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.504909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:57.505013 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.504956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:57.505456 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.505437    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-service-ca-bundle\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:57.507047 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.507025    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd34889-f57e-46ec-a1aa-6297a9d6c8e5-metrics-certs\") pod \"router-default-64bcb7459f-kdll9\" (UID: \"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5\") " pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:57.734714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.734671    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9ccvr\""
Jul 07 19:01:57.739999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.739979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:57.885603 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:57.885576    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64bcb7459f-kdll9"]
Jul 07 19:01:57.888913 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:01:57.888874    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd34889_f57e_46ec_a1aa_6297a9d6c8e5.slice/crio-b8c4fb272e8445ba961a4a5080715962164159dab1a44239a350b446b818b133 WatchSource:0}: Error finding container b8c4fb272e8445ba961a4a5080715962164159dab1a44239a350b446b818b133: Status 404 returned error can't find the container with id b8c4fb272e8445ba961a4a5080715962164159dab1a44239a350b446b818b133
Jul 07 19:01:58.729905 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:58.729871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64bcb7459f-kdll9" event={"ID":"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5","Type":"ContainerStarted","Data":"8e04e586d2417a0a69e933f4bdbbedc3bf7c43eebcb478f8b5e247d5a8f91d72"}
Jul 07 19:01:58.730347 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:58.729945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64bcb7459f-kdll9" event={"ID":"3cd34889-f57e-46ec-a1aa-6297a9d6c8e5","Type":"ContainerStarted","Data":"b8c4fb272e8445ba961a4a5080715962164159dab1a44239a350b446b818b133"}
Jul 07 19:01:58.740298 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:58.740270    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:58.742912 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:58.742892    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:58.751775 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:58.751721    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64bcb7459f-kdll9" podStartSLOduration=33.751687605 podStartE2EDuration="33.751687605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:58.751038078 +0000 UTC m=+84.470837292" watchObservedRunningTime="2026-07-07 19:01:58.751687605 +0000 UTC m=+84.471486795"
Jul 07 19:01:59.735087 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.735001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689" event={"ID":"a0647d78-e42e-4090-9592-9f551dbc8a4d","Type":"ContainerStarted","Data":"424c9e9398eb078af421ea02c2d1bff112e96f52785b61cf6ac05673e35e0eae"}
Jul 07 19:01:59.735087 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.735049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689" event={"ID":"a0647d78-e42e-4090-9592-9f551dbc8a4d","Type":"ContainerStarted","Data":"f4436930dc2086836c33f7b7341d1543c9c7aa5adc5ab34577c12f85358c578e"}
Jul 07 19:01:59.735571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.735184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:59.735571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.735222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689" event={"ID":"a0647d78-e42e-4090-9592-9f551dbc8a4d","Type":"ContainerStarted","Data":"febac5c75ee7c1729d0bfcbb413772eb49bc4c230bc2b8602e3459df4dfaa337"}
Jul 07 19:01:59.736515 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.736491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64bcb7459f-kdll9"
Jul 07 19:01:59.758989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:01:59.758948    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75778c7ccd-c2689" podStartSLOduration=1.4297011290000001 podStartE2EDuration="5.758936067s" totalImagesPullingTime="4.329234938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:59.758313653 +0000 UTC m=+85.478112845" watchObservedRunningTime="2026-07-07 19:01:59.758936067 +0000 UTC m=+85.478735269"
Jul 07 19:02:02.444406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:02.444373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:02:02.446628 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:02.446608    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97fde49a-d038-462a-bcee-cc86d2cec0ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qrdxq\" (UID: \"97fde49a-d038-462a-bcee-cc86d2cec0ec\") " pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:02:02.704553 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:02.704485    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\""
Jul 07 19:02:02.713023 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:02.713006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qrdxq"
Jul 07 19:02:02.836168 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:02.836138    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qrdxq"]
Jul 07 19:02:02.839008 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:02:02.838980    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fde49a_d038_462a_bcee_cc86d2cec0ec.slice/crio-149609e3603a3b8bb4c1cb4cba3a22c9a2e030ea684e3261edbbb0043f838600 WatchSource:0}: Error finding container 149609e3603a3b8bb4c1cb4cba3a22c9a2e030ea684e3261edbbb0043f838600: Status 404 returned error can't find the container with id 149609e3603a3b8bb4c1cb4cba3a22c9a2e030ea684e3261edbbb0043f838600
Jul 07 19:02:03.754661 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:03.754596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrdxq" event={"ID":"97fde49a-d038-462a-bcee-cc86d2cec0ec","Type":"ContainerStarted","Data":"149609e3603a3b8bb4c1cb4cba3a22c9a2e030ea684e3261edbbb0043f838600"}
Jul 07 19:02:06.765831 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:06.765803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrdxq" event={"ID":"97fde49a-d038-462a-bcee-cc86d2cec0ec","Type":"ContainerStarted","Data":"8e7122df821a51d9a67edef66f901ef8e80d8e52af151148edd73f4033054a14"}
Jul 07 19:02:07.771547 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:07.771504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrdxq" event={"ID":"97fde49a-d038-462a-bcee-cc86d2cec0ec","Type":"ContainerStarted","Data":"0e8597a59016f99a45e09372bba39e9a5d5f25e1cc2d166a5147369a607d5ac6"}
Jul 07 19:02:10.785743 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:10.785691    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qrdxq" event={"ID":"97fde49a-d038-462a-bcee-cc86d2cec0ec","Type":"ContainerStarted","Data":"ce19fb32ff13004619e7619af48f2419e408dfc6d687a1b9cf49d1859c44dfdd"}
Jul 07 19:02:10.805027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:10.804974    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qrdxq" podStartSLOduration=35.01104304 podStartE2EDuration="41.804959572s" totalImagesPullingTime="6.793916532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:10.804242285 +0000 UTC m=+96.524041473" watchObservedRunningTime="2026-07-07 19:02:10.804959572 +0000 UTC m=+96.524758778"
Jul 07 19:02:11.677578 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.677547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xdrgp"
Jul 07 19:02:11.784061 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.784035    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dbfg8"]
Jul 07 19:02:11.786685 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.786663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.789868 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.789848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 19:02:11.789998 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.789963    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 19:02:11.790123 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.790108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 19:02:11.790194 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.790109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jc4zb\""
Jul 07 19:02:11.790373 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.790354    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 19:02:11.790490 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.790475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 19:02:11.790763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.790748    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 19:02:11.919752 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.919864 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-textfile\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.919864 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-root\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.919991 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.919991 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-wtmp\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.919991 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw9hj\" (UniqueName: \"kubernetes.io/projected/42f797a1-6b65-4565-87ee-c9ee4f335ef3-kube-api-access-mw9hj\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.920135 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.919990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.920135 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.920014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-sys\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:11.920135 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:11.920089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-metrics-client-ca\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021072 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-root\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-wtmp\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw9hj\" (UniqueName: \"kubernetes.io/projected/42f797a1-6b65-4565-87ee-c9ee4f335ef3-kube-api-access-mw9hj\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021151    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-root\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-sys\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021236    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-wtmp\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-metrics-client-ca\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:02:12.021286    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-textfile\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021337    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f797a1-6b65-4565-87ee-c9ee4f335ef3-sys\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021434 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:02:12.021349    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls podName:42f797a1-6b65-4565-87ee-c9ee4f335ef3 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:12.521330532 +0000 UTC m=+98.241129708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls") pod "node-exporter-dbfg8" (UID: "42f797a1-6b65-4565-87ee-c9ee4f335ef3") : secret "node-exporter-tls" not found
Jul 07 19:02:12.021858 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021631    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-textfile\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021858 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021831    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.021947 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.021881    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f797a1-6b65-4565-87ee-c9ee4f335ef3-metrics-client-ca\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.023257 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.023236    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.032620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.032598    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw9hj\" (UniqueName: \"kubernetes.io/projected/42f797a1-6b65-4565-87ee-c9ee4f335ef3-kube-api-access-mw9hj\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.525179 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.525141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.527406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.527380    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42f797a1-6b65-4565-87ee-c9ee4f335ef3-node-exporter-tls\") pod \"node-exporter-dbfg8\" (UID: \"42f797a1-6b65-4565-87ee-c9ee4f335ef3\") " pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.695762 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.695735    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbfg8"
Jul 07 19:02:12.705925 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:02:12.705895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f797a1_6b65_4565_87ee_c9ee4f335ef3.slice/crio-66df652ebc6e7c3a71c133d5a2624fb1e571de322f0633f149d4fed5af927ca6 WatchSource:0}: Error finding container 66df652ebc6e7c3a71c133d5a2624fb1e571de322f0633f149d4fed5af927ca6: Status 404 returned error can't find the container with id 66df652ebc6e7c3a71c133d5a2624fb1e571de322f0633f149d4fed5af927ca6
Jul 07 19:02:12.793286 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:12.793217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbfg8" event={"ID":"42f797a1-6b65-4565-87ee-c9ee4f335ef3","Type":"ContainerStarted","Data":"66df652ebc6e7c3a71c133d5a2624fb1e571de322f0633f149d4fed5af927ca6"}
Jul 07 19:02:13.838238 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:13.838161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:02:13.840623 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:13.840594    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4393ec0-a42e-4579-a6a6-6bc959d3af3c-metrics-tls\") pod \"dns-default-zrz2k\" (UID: \"a4393ec0-a42e-4579-a6a6-6bc959d3af3c\") " pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:02:14.121355 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.121309    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\""
Jul 07 19:02:14.129002 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.128986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:02:14.250230 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.250195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrz2k"]
Jul 07 19:02:14.253885 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:02:14.253856    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4393ec0_a42e_4579_a6a6_6bc959d3af3c.slice/crio-c0ac7d62cafd817b03f8597b4a645d391425cbb72c17819720b4bffa49de7a11 WatchSource:0}: Error finding container c0ac7d62cafd817b03f8597b4a645d391425cbb72c17819720b4bffa49de7a11: Status 404 returned error can't find the container with id c0ac7d62cafd817b03f8597b4a645d391425cbb72c17819720b4bffa49de7a11
Jul 07 19:02:14.801695 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.801648    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrz2k" event={"ID":"a4393ec0-a42e-4579-a6a6-6bc959d3af3c","Type":"ContainerStarted","Data":"c0ac7d62cafd817b03f8597b4a645d391425cbb72c17819720b4bffa49de7a11"}
Jul 07 19:02:14.803571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.803540    2560 generic.go:356] "Generic (PLEG): container finished" podID="42f797a1-6b65-4565-87ee-c9ee4f335ef3" containerID="774996c716fb2c38daab027c0327bbfa768fb451eacfe35e961dac42a20c8c32" exitCode=0
Jul 07 19:02:14.803738 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:14.803631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbfg8" event={"ID":"42f797a1-6b65-4565-87ee-c9ee4f335ef3","Type":"ContainerDied","Data":"774996c716fb2c38daab027c0327bbfa768fb451eacfe35e961dac42a20c8c32"}
Jul 07 19:02:15.809169 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.809139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbfg8" event={"ID":"42f797a1-6b65-4565-87ee-c9ee4f335ef3","Type":"ContainerStarted","Data":"a6aae4f7d10fe6654e97b2a4c2ab31fa3045713d837dc688d5ca7e494a7acf2c"}
Jul 07 19:02:15.809556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.809197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbfg8" event={"ID":"42f797a1-6b65-4565-87ee-c9ee4f335ef3","Type":"ContainerStarted","Data":"04e1c39669e6d11731c400f46ef4b39f78301f442a410077b43747ae927313f2"}
Jul 07 19:02:15.810749 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.810724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrz2k" event={"ID":"a4393ec0-a42e-4579-a6a6-6bc959d3af3c","Type":"ContainerStarted","Data":"571f1f727b16f672eca4fc82d8dccff3848fb4ba6dcd9094fa8215fa3fbd0cb4"}
Jul 07 19:02:15.810859 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.810770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrz2k" event={"ID":"a4393ec0-a42e-4579-a6a6-6bc959d3af3c","Type":"ContainerStarted","Data":"adc7332492684ca2d10a0351da04aea13ac47612739f2bd2bee8ddc34205a32b"}
Jul 07 19:02:15.810859 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.810835    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:02:15.833425 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:15.833384    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dbfg8" podStartSLOduration=3.532145421 podStartE2EDuration="4.833366784s" totalImagesPullingTime="1.239398519s" totalInitContainerRuntime="61.822844ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:15.832131616 +0000 UTC m=+101.551930806" watchObservedRunningTime="2026-07-07 19:02:15.833366784 +0000 UTC m=+101.553165973"
Jul 07 19:02:23.596972 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.596922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zrz2k" podStartSLOduration=73.237817475 podStartE2EDuration="1m14.596907714s" totalImagesPullingTime="1.359090239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:15.851576415 +0000 UTC m=+101.571375605" watchObservedRunningTime="2026-07-07 19:02:23.596907714 +0000 UTC m=+109.316706904"
Jul 07 19:02:23.597486 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.597473    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:02:23.599990 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.599974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.602066 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 19:02:23.602182 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602162    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 19:02:23.602747 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602690    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 19:02:23.602862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602795    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 19:02:23.602862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602810    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 19:02:23.602862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602834    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6wjfm\""
Jul 07 19:02:23.602862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602838    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 19:02:23.602862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.602857    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 19:02:23.610478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.610454    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:02:23.708230 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.708331 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.708331 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnmb\" (UniqueName: \"kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.708412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.708412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.708412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.708393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808710 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.808936 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.808869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnmb\" (UniqueName: \"kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.809411 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.809384    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.809510 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.809440    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.809601 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.809577    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.811090 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.811066    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.811165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.811126    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.818399 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.818375    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnmb\" (UniqueName: \"kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb\") pod \"console-5656c88648-vq27d\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") " pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:23.913462 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:23.913419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:24.040848 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:24.040817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:02:24.043980 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:02:24.043943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c3cf9a_53af_45a2_a94d_ada8ee98fde2.slice/crio-6d98b2a68f499a85114609624f960d91d0fd46c7140790ba2dc359cca5907c5d WatchSource:0}: Error finding container 6d98b2a68f499a85114609624f960d91d0fd46c7140790ba2dc359cca5907c5d: Status 404 returned error can't find the container with id 6d98b2a68f499a85114609624f960d91d0fd46c7140790ba2dc359cca5907c5d
Jul 07 19:02:24.841999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:24.841965    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5656c88648-vq27d" event={"ID":"39c3cf9a-53af-45a2-a94d-ada8ee98fde2","Type":"ContainerStarted","Data":"6d98b2a68f499a85114609624f960d91d0fd46c7140790ba2dc359cca5907c5d"}
Jul 07 19:02:25.816160 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:25.816129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zrz2k"
Jul 07 19:02:27.855236 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:27.855190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5656c88648-vq27d" event={"ID":"39c3cf9a-53af-45a2-a94d-ada8ee98fde2","Type":"ContainerStarted","Data":"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"}
Jul 07 19:02:27.881205 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:27.881158    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5656c88648-vq27d" podStartSLOduration=1.697585549 podStartE2EDuration="4.881142372s" totalImagesPullingTime="3.183556823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:27.87984702 +0000 UTC m=+113.599646224" watchObservedRunningTime="2026-07-07 19:02:27.881142372 +0000 UTC m=+113.600941561"
Jul 07 19:02:33.913841 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:33.913805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:33.914303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:33.913862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:33.918508 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:33.918486    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:34.880860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:34.880819    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:02:44.048716 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.048678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:02:44.050955 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.050922    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed-metrics-certs\") pod \"network-metrics-daemon-pskjn\" (UID: \"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed\") " pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:02:44.128480 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.128455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\""
Jul 07 19:02:44.137258 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.137241    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pskjn"
Jul 07 19:02:44.278213 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.278191    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pskjn"]
Jul 07 19:02:44.280578 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:02:44.280550    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdc72d9_4def_47e0_94aa_c9cd3b0b06ed.slice/crio-07216f70554ca0a20654a6b14f13a7cc704ed75d783403fb446e393c53d48dc3 WatchSource:0}: Error finding container 07216f70554ca0a20654a6b14f13a7cc704ed75d783403fb446e393c53d48dc3: Status 404 returned error can't find the container with id 07216f70554ca0a20654a6b14f13a7cc704ed75d783403fb446e393c53d48dc3
Jul 07 19:02:44.913269 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:44.913230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pskjn" event={"ID":"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed","Type":"ContainerStarted","Data":"07216f70554ca0a20654a6b14f13a7cc704ed75d783403fb446e393c53d48dc3"}
Jul 07 19:02:45.084578 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:45.084549    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:02:45.918468 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:45.918440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pskjn" event={"ID":"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed","Type":"ContainerStarted","Data":"28fc6eea2ac5ef28463386878e71e25f9589dd88c58c3ee59811866acdf1873a"}
Jul 07 19:02:45.918612 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:45.918490    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pskjn" event={"ID":"5fdc72d9-4def-47e0-94aa-c9cd3b0b06ed","Type":"ContainerStarted","Data":"800c5499e65cf9626992059402e61d8e85a1c73fb9d26fedb176d8ab04b63f39"}
Jul 07 19:02:45.939189 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:45.939143    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pskjn" podStartSLOduration=129.843731268 podStartE2EDuration="2m10.939124103s" totalImagesPullingTime="1.095392835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:45.93853793 +0000 UTC m=+131.658337121" watchObservedRunningTime="2026-07-07 19:02:45.939124103 +0000 UTC m=+131.658923294"
Jul 07 19:02:59.966128 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:59.966097    2560 generic.go:356] "Generic (PLEG): container finished" podID="214887a9-0643-419b-80d2-ffd422948adc" containerID="79ccb1d7e86bbc99e168726c17e82545c07f7a664c645644add6f80b4e5f2b25" exitCode=0
Jul 07 19:02:59.966416 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:59.966171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd" event={"ID":"214887a9-0643-419b-80d2-ffd422948adc","Type":"ContainerDied","Data":"79ccb1d7e86bbc99e168726c17e82545c07f7a664c645644add6f80b4e5f2b25"}
Jul 07 19:02:59.966472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:02:59.966458    2560 scope.go:117] "RemoveContainer" containerID="79ccb1d7e86bbc99e168726c17e82545c07f7a664c645644add6f80b4e5f2b25"
Jul 07 19:03:00.970915 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:00.970884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f9qgd" event={"ID":"214887a9-0643-419b-80d2-ffd422948adc","Type":"ContainerStarted","Data":"32139416699eade854525441565122199d976c722fcdf74ed22159b2915ff9c8"}
Jul 07 19:03:10.108013 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.107953    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5656c88648-vq27d" podUID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" containerName="console" containerID="cri-o://a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e" gracePeriod=15
Jul 07 19:03:10.350942 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.350919    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:03:10.426964 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.426914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxnmb\" (UniqueName: \"kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.426964 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.426941    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.426964 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.426958    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.427145 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.426990    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.427145 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.427031    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.427145 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.427080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config\") pod \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\" (UID: \"39c3cf9a-53af-45a2-a94d-ada8ee98fde2\") "
Jul 07 19:03:10.427453 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.427425    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:03:10.427525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.427422    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca" (OuterVolumeSpecName: "service-ca") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:03:10.427525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.427473    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config" (OuterVolumeSpecName: "console-config") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:03:10.429062 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.429035    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:03:10.429130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.429068    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:03:10.429130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.429101    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb" (OuterVolumeSpecName: "kube-api-access-qxnmb") pod "39c3cf9a-53af-45a2-a94d-ada8ee98fde2" (UID: "39c3cf9a-53af-45a2-a94d-ada8ee98fde2"). InnerVolumeSpecName "kube-api-access-qxnmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:03:10.528474 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528453    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:10.528474 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528472    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnmb\" (UniqueName: \"kubernetes.io/projected/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-kube-api-access-qxnmb\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:10.528591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528483    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-oauth-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:10.528591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528491    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-console-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:10.528591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528500    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:10.528591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:10.528509    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39c3cf9a-53af-45a2-a94d-ada8ee98fde2-oauth-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:03:11.010571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.010524    2560 generic.go:356] "Generic (PLEG): container finished" podID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" containerID="a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e" exitCode=0
Jul 07 19:03:11.010571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.010567    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5656c88648-vq27d"
Jul 07 19:03:11.010832 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.010596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5656c88648-vq27d" event={"ID":"39c3cf9a-53af-45a2-a94d-ada8ee98fde2","Type":"ContainerDied","Data":"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"}
Jul 07 19:03:11.010832 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.010651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5656c88648-vq27d" event={"ID":"39c3cf9a-53af-45a2-a94d-ada8ee98fde2","Type":"ContainerDied","Data":"6d98b2a68f499a85114609624f960d91d0fd46c7140790ba2dc359cca5907c5d"}
Jul 07 19:03:11.010832 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.010672    2560 scope.go:117] "RemoveContainer" containerID="a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"
Jul 07 19:03:11.020043 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.020021    2560 scope.go:117] "RemoveContainer" containerID="a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"
Jul 07 19:03:11.020326 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:03:11.020302    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e\": container with ID starting with a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e not found: ID does not exist" containerID="a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"
Jul 07 19:03:11.020432 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.020333    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e"} err="failed to get container status \"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e\": rpc error: code = NotFound desc = could not find container \"a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e\": container with ID starting with a607defddb50379ad41b83c6673db2d08fcdc1638f2607ebbca7704fd5ea806e not found: ID does not exist"
Jul 07 19:03:11.046183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.046161    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:03:11.070177 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.070156    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5656c88648-vq27d"]
Jul 07 19:03:11.406841 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:11.406767    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" path="/var/lib/kubelet/pods/39c3cf9a-53af-45a2-a94d-ada8ee98fde2/volumes"
Jul 07 19:03:41.228831 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.228791    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:03:41.229337 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.229035    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" containerName="console"
Jul 07 19:03:41.229337 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.229047    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" containerName="console"
Jul 07 19:03:41.229337 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.229098    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c3cf9a-53af-45a2-a94d-ada8ee98fde2" containerName="console"
Jul 07 19:03:41.232292 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.232273    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.234529 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.234501    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 19:03:41.234645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.234515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6wjfm\""
Jul 07 19:03:41.235041 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.235020    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 19:03:41.235126 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.235106    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 19:03:41.235197 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.235179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 19:03:41.235682 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.235657    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 19:03:41.236554 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.236531    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 19:03:41.236748 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.236714    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 19:03:41.243293 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.243271    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 19:03:41.243764 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.243745    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:03:41.330905 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.330880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331000 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.330910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331000 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.330934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.331007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.331032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.331055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhbl\" (UniqueName: \"kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.331086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.331073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431606 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhbl\" (UniqueName: \"kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431976 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.431976 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.431742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.432459 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.432434    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.432563 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.432478    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.432563 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.432500    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.432678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.432649    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.434197 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.434177    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.434281 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.434229    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.444425 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.444406    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhbl\" (UniqueName: \"kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl\") pod \"console-7cdc48895c-9wt2q\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") " pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.543978 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.543953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:41.673100 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:41.673067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:03:41.676774 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:03:41.676740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf830da03_4a61_4e3f_887f_2605daa8d389.slice/crio-8a946232463210a870c48f6b4561b6642b080372bb4063526f30b1685c1aa3e3 WatchSource:0}: Error finding container 8a946232463210a870c48f6b4561b6642b080372bb4063526f30b1685c1aa3e3: Status 404 returned error can't find the container with id 8a946232463210a870c48f6b4561b6642b080372bb4063526f30b1685c1aa3e3
Jul 07 19:03:42.122202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:42.122166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc48895c-9wt2q" event={"ID":"f830da03-4a61-4e3f-887f-2605daa8d389","Type":"ContainerStarted","Data":"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"}
Jul 07 19:03:42.122368 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:42.122217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc48895c-9wt2q" event={"ID":"f830da03-4a61-4e3f-887f-2605daa8d389","Type":"ContainerStarted","Data":"8a946232463210a870c48f6b4561b6642b080372bb4063526f30b1685c1aa3e3"}
Jul 07 19:03:42.150691 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:42.150644    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cdc48895c-9wt2q" podStartSLOduration=1.150628787 podStartE2EDuration="1.150628787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:42.150561777 +0000 UTC m=+187.870360981" watchObservedRunningTime="2026-07-07 19:03:42.150628787 +0000 UTC m=+187.870427978"
Jul 07 19:03:51.544480 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:51.544434    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:51.544947 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:51.544494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:51.549454 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:51.549426    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:03:52.164188 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:03:52.164152    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:05:20.094077 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.094035    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:05:20.097175 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.097153    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.165845 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.165815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.165970 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.165853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.165970 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.165874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.165970 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.165889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv8pn\" (UniqueName: \"kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.166127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.165982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.166127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.166031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.166127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.166060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.174952 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.174929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:05:20.266796 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.266772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.266910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.266806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.266910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.266831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.266990 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.266937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv8pn\" (UniqueName: \"kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.266990 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.266976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.267088 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.267004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.267088 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.267020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.267570 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.267545    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.267688 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.267550    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.267688 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.267660    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.268071 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.268050    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.269280 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.269253    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.269494 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.269477    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.276459 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.276438    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv8pn\" (UniqueName: \"kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn\") pod \"console-89cc6879f-t7s8d\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") " pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.407190 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.406971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:20.531907 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:20.531873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:05:20.534695 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:05:20.534666    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0518f22_2776_46b8_b01c_03cf0bdcdc85.slice/crio-396bdffcc61a3486a8937f6c49e3236f4910882b10f908f9fa19dd1c372f386b WatchSource:0}: Error finding container 396bdffcc61a3486a8937f6c49e3236f4910882b10f908f9fa19dd1c372f386b: Status 404 returned error can't find the container with id 396bdffcc61a3486a8937f6c49e3236f4910882b10f908f9fa19dd1c372f386b
Jul 07 19:05:21.457873 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:21.457828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89cc6879f-t7s8d" event={"ID":"c0518f22-2776-46b8-b01c-03cf0bdcdc85","Type":"ContainerStarted","Data":"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"}
Jul 07 19:05:21.458243 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:21.457892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89cc6879f-t7s8d" event={"ID":"c0518f22-2776-46b8-b01c-03cf0bdcdc85","Type":"ContainerStarted","Data":"396bdffcc61a3486a8937f6c49e3236f4910882b10f908f9fa19dd1c372f386b"}
Jul 07 19:05:21.478436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:21.478369    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-89cc6879f-t7s8d" podStartSLOduration=1.478352354 podStartE2EDuration="1.478352354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:05:21.476739187 +0000 UTC m=+287.196538377" watchObservedRunningTime="2026-07-07 19:05:21.478352354 +0000 UTC m=+287.198151543"
Jul 07 19:05:30.407751 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:30.407696    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:30.408146 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:30.407781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:30.412475 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:30.412453    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:30.494114 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:30.494091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:05:30.550317 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:30.550287    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:05:35.195617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:35.195583    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 19:05:55.571643 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.571589    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7cdc48895c-9wt2q" podUID="f830da03-4a61-4e3f-887f-2605daa8d389" containerName="console" containerID="cri-o://15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc" gracePeriod=15
Jul 07 19:05:55.808563 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.808542    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:05:55.914286 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914212    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914286 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914246    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914286 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckhbl\" (UniqueName: \"kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914540 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914304    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914540 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914540 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914540 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert\") pod \"f830da03-4a61-4e3f-887f-2605daa8d389\" (UID: \"f830da03-4a61-4e3f-887f-2605daa8d389\") "
Jul 07 19:05:55.914780 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914649    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config" (OuterVolumeSpecName: "console-config") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:05:55.914780 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914722    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca" (OuterVolumeSpecName: "service-ca") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:05:55.914967 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.914940    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:05:55.915072 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.915043    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:05:55.916406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.916379    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:05:55.916507 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.916433    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl" (OuterVolumeSpecName: "kube-api-access-ckhbl") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "kube-api-access-ckhbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:05:55.916507 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:55.916485    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f830da03-4a61-4e3f-887f-2605daa8d389" (UID: "f830da03-4a61-4e3f-887f-2605daa8d389"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:05:56.014909 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014879    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-trusted-ca-bundle\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.014909 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014908    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-console-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.014909 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014919    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhbl\" (UniqueName: \"kubernetes.io/projected/f830da03-4a61-4e3f-887f-2605daa8d389-kube-api-access-ckhbl\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.015069 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014929    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.015069 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014939    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f830da03-4a61-4e3f-887f-2605daa8d389-console-oauth-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.015069 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014947    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.015069 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.014955    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f830da03-4a61-4e3f-887f-2605daa8d389-oauth-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:05:56.581000 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.580968    2560 generic.go:356] "Generic (PLEG): container finished" podID="f830da03-4a61-4e3f-887f-2605daa8d389" containerID="15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc" exitCode=0
Jul 07 19:05:56.581370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.581016    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdc48895c-9wt2q"
Jul 07 19:05:56.581370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.581056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc48895c-9wt2q" event={"ID":"f830da03-4a61-4e3f-887f-2605daa8d389","Type":"ContainerDied","Data":"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"}
Jul 07 19:05:56.581370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.581107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdc48895c-9wt2q" event={"ID":"f830da03-4a61-4e3f-887f-2605daa8d389","Type":"ContainerDied","Data":"8a946232463210a870c48f6b4561b6642b080372bb4063526f30b1685c1aa3e3"}
Jul 07 19:05:56.581370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.581127    2560 scope.go:117] "RemoveContainer" containerID="15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"
Jul 07 19:05:56.589283 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.589268    2560 scope.go:117] "RemoveContainer" containerID="15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"
Jul 07 19:05:56.589525 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:05:56.589506    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc\": container with ID starting with 15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc not found: ID does not exist" containerID="15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"
Jul 07 19:05:56.589577 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.589530    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc"} err="failed to get container status \"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc\": rpc error: code = NotFound desc = could not find container \"15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc\": container with ID starting with 15a954ecb25a69e22bf73ed5e8c9c529e3674fcff0a5dea8bfd7b12ea6515fcc not found: ID does not exist"
Jul 07 19:05:56.606095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.606075    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:05:56.610481 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:56.610462    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cdc48895c-9wt2q"]
Jul 07 19:05:57.406801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:05:57.406761    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f830da03-4a61-4e3f-887f-2605daa8d389" path="/var/lib/kubelet/pods/f830da03-4a61-4e3f-887f-2605daa8d389/volumes"
Jul 07 19:06:01.423883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.423853    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fkrr8"]
Jul 07 19:06:01.424238 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.424092    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f830da03-4a61-4e3f-887f-2605daa8d389" containerName="console"
Jul 07 19:06:01.424238 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.424104    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f830da03-4a61-4e3f-887f-2605daa8d389" containerName="console"
Jul 07 19:06:01.424238 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.424159    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f830da03-4a61-4e3f-887f-2605daa8d389" containerName="console"
Jul 07 19:06:01.427071 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.427057    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.430616 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.430598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 19:06:01.430767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.430633    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 19:06:01.431070 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.431054    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wbd5l\""
Jul 07 19:06:01.437952 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.437932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fkrr8"]
Jul 07 19:06:01.552412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.552387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.552533 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.552428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtll\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-kube-api-access-qbtll\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.653095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.653063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.653215 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.653119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtll\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-kube-api-access-qbtll\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.662966 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.662942    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtll\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-kube-api-access-qbtll\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.663966 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.663949    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34909cff-2917-4c2f-8a60-386cf2b5931a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fkrr8\" (UID: \"34909cff-2917-4c2f-8a60-386cf2b5931a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.735179 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.735125    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:01.856844 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.856820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fkrr8"]
Jul 07 19:06:01.859447 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:06:01.859419    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34909cff_2917_4c2f_8a60_386cf2b5931a.slice/crio-c26f543d1cd4b81809f056d10057089af5810a24c64336bf48451e96a7f3bde3 WatchSource:0}: Error finding container c26f543d1cd4b81809f056d10057089af5810a24c64336bf48451e96a7f3bde3: Status 404 returned error can't find the container with id c26f543d1cd4b81809f056d10057089af5810a24c64336bf48451e96a7f3bde3
Jul 07 19:06:01.861817 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:01.861800    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:06:02.605154 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:02.605110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8" event={"ID":"34909cff-2917-4c2f-8a60-386cf2b5931a","Type":"ContainerStarted","Data":"c26f543d1cd4b81809f056d10057089af5810a24c64336bf48451e96a7f3bde3"}
Jul 07 19:06:05.618547 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:05.618509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8" event={"ID":"34909cff-2917-4c2f-8a60-386cf2b5931a","Type":"ContainerStarted","Data":"35d3f650a21614221cfb930a63eba1c3b88e0dcb825560ea6235d41ef840378e"}
Jul 07 19:06:05.618931 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:05.618559    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:11.623897 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:11.623866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8"
Jul 07 19:06:11.655498 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:11.655450    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fkrr8" podStartSLOduration=7.859852079 podStartE2EDuration="10.655436309s" totalImagesPullingTime="2.79558423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:05.685887198 +0000 UTC m=+331.405686388" watchObservedRunningTime="2026-07-07 19:06:11.655436309 +0000 UTC m=+337.375235500"
Jul 07 19:06:29.880638 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.880569    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"]
Jul 07 19:06:29.882497 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.882483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:29.887004 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.886979    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 19:06:29.887142 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.886984    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 19:06:29.887142 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.886988    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 19:06:29.887347 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.887318    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vlfr\""
Jul 07 19:06:29.887417 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.887375    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 19:06:29.903470 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:29.903450    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"]
Jul 07 19:06:30.039511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.039478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.039678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.039519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbmr\" (UniqueName: \"kubernetes.io/projected/47f799f5-fba3-4227-8974-258b4d93543d-kube-api-access-hdbmr\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.039678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.039552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.039678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.039569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.039678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.039590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.140676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.140600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.140676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.140644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.140676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.140666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.140877 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.140719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.140877 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.140740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbmr\" (UniqueName: \"kubernetes.io/projected/47f799f5-fba3-4227-8974-258b4d93543d-kube-api-access-hdbmr\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.141148 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.141115    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.142855 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.142830    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f799f5-fba3-4227-8974-258b4d93543d-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.143199 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.143175    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.143199 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.143191    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f799f5-fba3-4227-8974-258b4d93543d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.158681 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.158657    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbmr\" (UniqueName: \"kubernetes.io/projected/47f799f5-fba3-4227-8974-258b4d93543d-kube-api-access-hdbmr\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-5rzkk\" (UID: \"47f799f5-fba3-4227-8974-258b4d93543d\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.192590 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.192569    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:30.327299 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.327267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"]
Jul 07 19:06:30.330238 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:06:30.330213    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f799f5_fba3_4227_8974_258b4d93543d.slice/crio-a85c24a9c0ed1b23a7f0c14461575bf7f0fb7b4d50f970918dbf68e6eebcdcac WatchSource:0}: Error finding container a85c24a9c0ed1b23a7f0c14461575bf7f0fb7b4d50f970918dbf68e6eebcdcac: Status 404 returned error can't find the container with id a85c24a9c0ed1b23a7f0c14461575bf7f0fb7b4d50f970918dbf68e6eebcdcac
Jul 07 19:06:30.702731 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:30.702681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk" event={"ID":"47f799f5-fba3-4227-8974-258b4d93543d","Type":"ContainerStarted","Data":"a85c24a9c0ed1b23a7f0c14461575bf7f0fb7b4d50f970918dbf68e6eebcdcac"}
Jul 07 19:06:33.717675 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:33.717602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk" event={"ID":"47f799f5-fba3-4227-8974-258b4d93543d","Type":"ContainerStarted","Data":"3807a7c2a46919c109bb73792dab94a8071e787fa0283aeb8a059ee82577732f"}
Jul 07 19:06:34.723092 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:34.723008    2560 generic.go:356] "Generic (PLEG): container finished" podID="47f799f5-fba3-4227-8974-258b4d93543d" containerID="3807a7c2a46919c109bb73792dab94a8071e787fa0283aeb8a059ee82577732f" exitCode=0
Jul 07 19:06:34.723481 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:34.723127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk" event={"ID":"47f799f5-fba3-4227-8974-258b4d93543d","Type":"ContainerDied","Data":"3807a7c2a46919c109bb73792dab94a8071e787fa0283aeb8a059ee82577732f"}
Jul 07 19:06:35.729477 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:35.729443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk" event={"ID":"47f799f5-fba3-4227-8974-258b4d93543d","Type":"ContainerStarted","Data":"4648696c06f28a180979e8c7cb93d8ed12ee3ca83b9938ce8385e44409a457cb"}
Jul 07 19:06:35.730011 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:35.729594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:35.793778 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:35.793729    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk" podStartSLOduration=2.461552121 podStartE2EDuration="6.793710663s" totalImagesPullingTime="3.30552357s" totalInitContainerRuntime="1.026634972s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:35.792933255 +0000 UTC m=+361.512732444" watchObservedRunningTime="2026-07-07 19:06:35.793710663 +0000 UTC m=+361.513509845"
Jul 07 19:06:46.735278 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:46.735245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-5rzkk"
Jul 07 19:06:49.846936 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.846902    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"]
Jul 07 19:06:49.853493 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.853476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.857720 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.857687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 19:06:49.858322 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.858299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ftmsk\""
Jul 07 19:06:49.858414 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.858336    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 19:06:49.874965 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.874938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"]
Jul 07 19:06:49.881686 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.881666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a247b30-5bff-4f74-b342-9bb8cb05215a-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.881794 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.881733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvr8\" (UniqueName: \"kubernetes.io/projected/3a247b30-5bff-4f74-b342-9bb8cb05215a-kube-api-access-mxvr8\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.881794 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.881790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a247b30-5bff-4f74-b342-9bb8cb05215a-tmp\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.982073 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.982048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvr8\" (UniqueName: \"kubernetes.io/projected/3a247b30-5bff-4f74-b342-9bb8cb05215a-kube-api-access-mxvr8\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.982204 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.982089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a247b30-5bff-4f74-b342-9bb8cb05215a-tmp\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.982204 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.982130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a247b30-5bff-4f74-b342-9bb8cb05215a-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.984393 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.984363    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a247b30-5bff-4f74-b342-9bb8cb05215a-tmp\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.984538 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.984517    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a247b30-5bff-4f74-b342-9bb8cb05215a-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:49.998061 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:49.998039    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvr8\" (UniqueName: \"kubernetes.io/projected/3a247b30-5bff-4f74-b342-9bb8cb05215a-kube-api-access-mxvr8\") pod \"kube-auth-proxy-7cd4df5c96-2cfpr\" (UID: \"3a247b30-5bff-4f74-b342-9bb8cb05215a\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:50.163181 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:50.163096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"
Jul 07 19:06:50.309813 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:50.309784    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr"]
Jul 07 19:06:50.312087 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:06:50.312054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a247b30_5bff_4f74_b342_9bb8cb05215a.slice/crio-aaab3e5aaca6e78c6b27abfb98e9e988dde46c489ee0b045d902b9ad00dc5ef4 WatchSource:0}: Error finding container aaab3e5aaca6e78c6b27abfb98e9e988dde46c489ee0b045d902b9ad00dc5ef4: Status 404 returned error can't find the container with id aaab3e5aaca6e78c6b27abfb98e9e988dde46c489ee0b045d902b9ad00dc5ef4
Jul 07 19:06:50.782652 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:50.782618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr" event={"ID":"3a247b30-5bff-4f74-b342-9bb8cb05215a","Type":"ContainerStarted","Data":"aaab3e5aaca6e78c6b27abfb98e9e988dde46c489ee0b045d902b9ad00dc5ef4"}
Jul 07 19:06:51.509861 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.509826    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n7662"]
Jul 07 19:06:51.511833 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.511817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.516196 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.516172    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bnx9h\""
Jul 07 19:06:51.517793 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.517758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 07 19:06:51.529946 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.529911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n7662"]
Jul 07 19:06:51.594856 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.594833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqn7\" (UniqueName: \"kubernetes.io/projected/42730f4f-2eac-4edf-aad2-73f7ccfc9593-kube-api-access-vzqn7\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.594974 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.594870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42730f4f-2eac-4edf-aad2-73f7ccfc9593-tls-certs\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.695678 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.695646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqn7\" (UniqueName: \"kubernetes.io/projected/42730f4f-2eac-4edf-aad2-73f7ccfc9593-kube-api-access-vzqn7\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.695854 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.695686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42730f4f-2eac-4edf-aad2-73f7ccfc9593-tls-certs\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.698097 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.698063    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42730f4f-2eac-4edf-aad2-73f7ccfc9593-tls-certs\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.709176 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.709151    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqn7\" (UniqueName: \"kubernetes.io/projected/42730f4f-2eac-4edf-aad2-73f7ccfc9593-kube-api-access-vzqn7\") pod \"model-serving-api-54d89bd5fc-n7662\" (UID: \"42730f4f-2eac-4edf-aad2-73f7ccfc9593\") " pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.822296 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.822221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:51.984721 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:51.984676    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n7662"]
Jul 07 19:06:51.987589 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:06:51.987549    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42730f4f_2eac_4edf_aad2_73f7ccfc9593.slice/crio-a6f0d67563ab4642a4ecc384ea5a7407aa28360ce2c5a742de9a53eae5f8e151 WatchSource:0}: Error finding container a6f0d67563ab4642a4ecc384ea5a7407aa28360ce2c5a742de9a53eae5f8e151: Status 404 returned error can't find the container with id a6f0d67563ab4642a4ecc384ea5a7407aa28360ce2c5a742de9a53eae5f8e151
Jul 07 19:06:52.182362 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.182281    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:06:52.185505 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.185484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.188628 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.188606    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 19:06:52.188628 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.188619    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hvtm8\""
Jul 07 19:06:52.189041 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.189026    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 19:06:52.190131 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.190115    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 19:06:52.190131 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.190122    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 19:06:52.199953 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.199934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.199971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.199992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200121 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.200051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200121 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.200081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200180 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.200137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwk6\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.200216 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.200176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.255797 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.255763    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:06:52.300662 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300992 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwk6\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.300992 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.300824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.301496 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.301470    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.303046 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.303023    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.303217 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.303200    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.303611 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.303588    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.303670 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.303597    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.311914 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.311890    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwk6\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.312222 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.312204    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-vqz2v\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.496585 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.496521    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:52.665432 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.665354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:06:52.668321 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:06:52.668289    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4039a2ab_3910_4822_9996_57a54fcddfb5.slice/crio-feadbd4242722837146260cc1ef8a12178a91b76b91eaa90364152f53478dda7 WatchSource:0}: Error finding container feadbd4242722837146260cc1ef8a12178a91b76b91eaa90364152f53478dda7: Status 404 returned error can't find the container with id feadbd4242722837146260cc1ef8a12178a91b76b91eaa90364152f53478dda7
Jul 07 19:06:52.793507 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.793422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" event={"ID":"4039a2ab-3910-4822-9996-57a54fcddfb5","Type":"ContainerStarted","Data":"feadbd4242722837146260cc1ef8a12178a91b76b91eaa90364152f53478dda7"}
Jul 07 19:06:52.795220 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:52.795186    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n7662" event={"ID":"42730f4f-2eac-4edf-aad2-73f7ccfc9593","Type":"ContainerStarted","Data":"a6f0d67563ab4642a4ecc384ea5a7407aa28360ce2c5a742de9a53eae5f8e151"}
Jul 07 19:06:55.747241 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:55.747209    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:06:55.747593 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:55.747270    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:06:55.813002 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:55.812969    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n7662" event={"ID":"42730f4f-2eac-4edf-aad2-73f7ccfc9593","Type":"ContainerStarted","Data":"4aa2b0166f486b385e1ac73d4723afb8d3513e5e0361a4d1c9ca47a1eadf8b36"}
Jul 07 19:06:55.813125 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:55.813107    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:06:55.839629 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:55.837348    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-n7662" podStartSLOduration=1.13617435 podStartE2EDuration="4.837329812s" totalImagesPullingTime="3.701155462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:55.834798512 +0000 UTC m=+381.554597702" watchObservedRunningTime="2026-07-07 19:06:55.837329812 +0000 UTC m=+381.557129000"
Jul 07 19:06:56.819498 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:56.819455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr" event={"ID":"3a247b30-5bff-4f74-b342-9bb8cb05215a","Type":"ContainerStarted","Data":"eef5fd7a48d65680119c204a55d33c6d9d6825cdf223e03b9cf17e8c5864fc52"}
Jul 07 19:06:56.821340 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:56.821308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" event={"ID":"4039a2ab-3910-4822-9996-57a54fcddfb5","Type":"ContainerStarted","Data":"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"}
Jul 07 19:06:56.849439 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:56.849400    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-2cfpr" podStartSLOduration=2.466044448 podStartE2EDuration="7.84938526s" totalImagesPullingTime="5.383340812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:56.848352832 +0000 UTC m=+382.568152024" watchObservedRunningTime="2026-07-07 19:06:56.84938526 +0000 UTC m=+382.569184453"
Jul 07 19:06:56.871555 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:56.871509    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" podStartSLOduration=1.795742025 podStartE2EDuration="4.871494645s" totalImagesPullingTime="3.07575262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:56.870985223 +0000 UTC m=+382.590784425" watchObservedRunningTime="2026-07-07 19:06:56.871494645 +0000 UTC m=+382.591293835"
Jul 07 19:06:57.826110 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:57.826070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:57.827927 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:57.827902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:06:59.764695 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.764660    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"]
Jul 07 19:06:59.766923 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.766905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.786612 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.786588    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"]
Jul 07 19:06:59.866308 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866438 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866438 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866438 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfc8w\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-kube-api-access-hfc8w\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866584 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866584 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.866646 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.866595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/301ff9ce-2e49-4b96-8586-642c134645c7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967536 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967650 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967650 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/301ff9ce-2e49-4b96-8586-642c134645c7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967650 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967821 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967821 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.967982 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.967831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc8w\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-kube-api-access-hfc8w\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.968476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.968455    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.969943 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.969916    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.970037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.969998    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.970202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.970179    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/301ff9ce-2e49-4b96-8586-642c134645c7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.970513 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.970496    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/301ff9ce-2e49-4b96-8586-642c134645c7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.979147 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.979129    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:06:59.980858 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:06:59.980837    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc8w\" (UniqueName: \"kubernetes.io/projected/301ff9ce-2e49-4b96-8586-642c134645c7-kube-api-access-hfc8w\") pod \"istiod-openshift-gateway-94bb8fbfd-tfsk8\" (UID: \"301ff9ce-2e49-4b96-8586-642c134645c7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:07:00.075281 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.075238    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:07:00.336079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.336006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"]
Jul 07 19:07:00.346284 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:07:00.346257    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301ff9ce_2e49_4b96_8586_642c134645c7.slice/crio-49ef8fcc8fcf2876979e31fdbdcc5ce64766bd3e059f6e901ae496fc5fcc16a1 WatchSource:0}: Error finding container 49ef8fcc8fcf2876979e31fdbdcc5ce64766bd3e059f6e901ae496fc5fcc16a1: Status 404 returned error can't find the container with id 49ef8fcc8fcf2876979e31fdbdcc5ce64766bd3e059f6e901ae496fc5fcc16a1
Jul 07 19:07:00.349179 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.349144    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:07:00.349262 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.349201    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:07:00.841478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.841431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8" event={"ID":"301ff9ce-2e49-4b96-8586-642c134645c7","Type":"ContainerStarted","Data":"60b6b2ea1b3e817b85d6799ce46409965d1ede03c54f210ef007b8c16c26b6cd"}
Jul 07 19:07:00.841478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.841489    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8" event={"ID":"301ff9ce-2e49-4b96-8586-642c134645c7","Type":"ContainerStarted","Data":"49ef8fcc8fcf2876979e31fdbdcc5ce64766bd3e059f6e901ae496fc5fcc16a1"}
Jul 07 19:07:00.842010 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:00.841517    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:07:01.847288 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:01.847256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8"
Jul 07 19:07:01.899629 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:01.899574    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tfsk8" podStartSLOduration=2.8995557659999998 podStartE2EDuration="2.899555766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:00.888938377 +0000 UTC m=+386.608737570" watchObservedRunningTime="2026-07-07 19:07:01.899555766 +0000 UTC m=+387.619354958"
Jul 07 19:07:01.966776 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:01.966748    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:07:01.966944 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:01.966923    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" podUID="4039a2ab-3910-4822-9996-57a54fcddfb5" containerName="discovery" containerID="cri-o://627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c" gracePeriod=30
Jul 07 19:07:02.215413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.215385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:07:02.288472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgwk6\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288606 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288483    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288606 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288606 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288532    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288606 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.288860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.288642    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap\") pod \"4039a2ab-3910-4822-9996-57a54fcddfb5\" (UID: \"4039a2ab-3910-4822-9996-57a54fcddfb5\") "
Jul 07 19:07:02.289125 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.289076    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:07:02.290939 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.290915    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6" (OuterVolumeSpecName: "kube-api-access-rgwk6") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "kube-api-access-rgwk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:07:02.291011 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.290964    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:07:02.291124 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.291106    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:07:02.291178 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.291146    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs" (OuterVolumeSpecName: "local-certs") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:07:02.291213 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.291146    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:07:02.291213 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.291182    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts" (OuterVolumeSpecName: "cacerts") pod "4039a2ab-3910-4822-9996-57a54fcddfb5" (UID: "4039a2ab-3910-4822-9996-57a54fcddfb5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:07:02.389462 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389430    2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4039a2ab-3910-4822-9996-57a54fcddfb5-local-certs\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389462 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389458    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-token\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389470    2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-cacerts\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389481    2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-kubeconfig\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389493    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-dns-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389504    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4039a2ab-3910-4822-9996-57a54fcddfb5-istio-csr-ca-configmap\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.389614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.389517    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwk6\" (UniqueName: \"kubernetes.io/projected/4039a2ab-3910-4822-9996-57a54fcddfb5-kube-api-access-rgwk6\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:07:02.851556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.851521    2560 generic.go:356] "Generic (PLEG): container finished" podID="4039a2ab-3910-4822-9996-57a54fcddfb5" containerID="627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c" exitCode=0
Jul 07 19:07:02.851946 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.851565    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"
Jul 07 19:07:02.851946 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.851609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" event={"ID":"4039a2ab-3910-4822-9996-57a54fcddfb5","Type":"ContainerDied","Data":"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"}
Jul 07 19:07:02.851946 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.851660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v" event={"ID":"4039a2ab-3910-4822-9996-57a54fcddfb5","Type":"ContainerDied","Data":"feadbd4242722837146260cc1ef8a12178a91b76b91eaa90364152f53478dda7"}
Jul 07 19:07:02.851946 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.851682    2560 scope.go:117] "RemoveContainer" containerID="627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"
Jul 07 19:07:02.859941 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.859927    2560 scope.go:117] "RemoveContainer" containerID="627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"
Jul 07 19:07:02.860195 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:07:02.860175    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c\": container with ID starting with 627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c not found: ID does not exist" containerID="627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"
Jul 07 19:07:02.860233 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.860206    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c"} err="failed to get container status \"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c\": rpc error: code = NotFound desc = could not find container \"627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c\": container with ID starting with 627abe72dd8c9177722a36162127922e17fe8ab7706f8c28020ee864d925cb9c not found: ID does not exist"
Jul 07 19:07:02.881144 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.881124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:07:02.882358 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:02.882340    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vqz2v"]
Jul 07 19:07:03.407748 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.407690    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4039a2ab-3910-4822-9996-57a54fcddfb5" path="/var/lib/kubelet/pods/4039a2ab-3910-4822-9996-57a54fcddfb5/volumes"
Jul 07 19:07:03.408072 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.408045    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"]
Jul 07 19:07:03.408519 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.408492    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4039a2ab-3910-4822-9996-57a54fcddfb5" containerName="discovery"
Jul 07 19:07:03.408519 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.408519    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4039a2ab-3910-4822-9996-57a54fcddfb5" containerName="discovery"
Jul 07 19:07:03.408696 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.408609    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4039a2ab-3910-4822-9996-57a54fcddfb5" containerName="discovery"
Jul 07 19:07:03.411246 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.411222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.415213 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.415196    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 19:07:03.415356 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.415343    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4f69c\""
Jul 07 19:07:03.441587 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.441562    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"]
Jul 07 19:07:03.499341 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.499320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2zg\" (UniqueName: \"kubernetes.io/projected/71a5782b-194e-4a47-b998-52ceadacac51-kube-api-access-ws2zg\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.499439 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.499357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71a5782b-194e-4a47-b998-52ceadacac51-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.599663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.599636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2zg\" (UniqueName: \"kubernetes.io/projected/71a5782b-194e-4a47-b998-52ceadacac51-kube-api-access-ws2zg\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.599817 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.599693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71a5782b-194e-4a47-b998-52ceadacac51-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.601843 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.601821    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71a5782b-194e-4a47-b998-52ceadacac51-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.616566 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.616533    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2zg\" (UniqueName: \"kubernetes.io/projected/71a5782b-194e-4a47-b998-52ceadacac51-kube-api-access-ws2zg\") pod \"llmisvc-controller-manager-777c6bd7d5-vhk5x\" (UID: \"71a5782b-194e-4a47-b998-52ceadacac51\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.720325 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.720272    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:03.854371 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:03.854331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"]
Jul 07 19:07:03.857636 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:07:03.857611    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod71a5782b_194e_4a47_b998_52ceadacac51.slice/crio-40006186ac92fdf9cda415b38b9de090ee108c4a57b1d11f2c39fd25fbdd544d WatchSource:0}: Error finding container 40006186ac92fdf9cda415b38b9de090ee108c4a57b1d11f2c39fd25fbdd544d: Status 404 returned error can't find the container with id 40006186ac92fdf9cda415b38b9de090ee108c4a57b1d11f2c39fd25fbdd544d
Jul 07 19:07:04.862416 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:04.862382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x" event={"ID":"71a5782b-194e-4a47-b998-52ceadacac51","Type":"ContainerStarted","Data":"40006186ac92fdf9cda415b38b9de090ee108c4a57b1d11f2c39fd25fbdd544d"}
Jul 07 19:07:06.826996 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:06.826967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-n7662"
Jul 07 19:07:07.878975 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:07.878934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x" event={"ID":"71a5782b-194e-4a47-b998-52ceadacac51","Type":"ContainerStarted","Data":"3f003181d6dba36bb6641aaf1c16fde06938ee2a3ecca7107deb87e6efd18992"}
Jul 07 19:07:07.879507 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:07.879055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:07:07.916684 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:07.916631    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x" podStartSLOduration=1.399957363 podStartE2EDuration="4.916618456s" totalImagesPullingTime="3.516661093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:07.915136293 +0000 UTC m=+393.634935480" watchObservedRunningTime="2026-07-07 19:07:07.916618456 +0000 UTC m=+393.636417712"
Jul 07 19:07:38.885130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:07:38.885099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhk5x"
Jul 07 19:08:24.411851 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.411813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"]
Jul 07 19:08:24.414503 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.414478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.419497 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.419473    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 19:08:24.419617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.419594    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 19:08:24.419668 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.419614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pxtsn\""
Jul 07 19:08:24.432131 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.432108    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"]
Jul 07 19:08:24.475108 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.475082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sln8c\" (UniqueName: \"kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.475225 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.475134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.575894 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.575868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sln8c\" (UniqueName: \"kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.576003 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.575900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.576190 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.576173    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.594232 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.594211    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sln8c\" (UniqueName: \"kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.724467 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.724402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:24.863419 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:24.863380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"]
Jul 07 19:08:24.865826 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:08:24.865799    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348357bd_0dd0_421f_bf5a_9ab237730434.slice/crio-e19508e0410ab643d4fe7e8dfc133cfaeba4b78641ed8aff0dc212e646acf33f WatchSource:0}: Error finding container e19508e0410ab643d4fe7e8dfc133cfaeba4b78641ed8aff0dc212e646acf33f: Status 404 returned error can't find the container with id e19508e0410ab643d4fe7e8dfc133cfaeba4b78641ed8aff0dc212e646acf33f
Jul 07 19:08:25.203792 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:25.203760    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9" event={"ID":"348357bd-0dd0-421f-bf5a-9ab237730434","Type":"ContainerStarted","Data":"e19508e0410ab643d4fe7e8dfc133cfaeba4b78641ed8aff0dc212e646acf33f"}
Jul 07 19:08:29.223172 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:29.223134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9" event={"ID":"348357bd-0dd0-421f-bf5a-9ab237730434","Type":"ContainerStarted","Data":"ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0"}
Jul 07 19:08:29.223549 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:29.223349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:29.263475 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:29.263424    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9" podStartSLOduration=1.389480964 podStartE2EDuration="5.263410252s" totalImagesPullingTime="3.873929288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:29.26227105 +0000 UTC m=+474.982070239" watchObservedRunningTime="2026-07-07 19:08:29.263410252 +0000 UTC m=+474.983209442"
Jul 07 19:08:40.229401 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:40.229359    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:42.040115 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.040075    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"]
Jul 07 19:08:42.040580 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.040311    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" containerName="manager" containerID="cri-o://ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0" gracePeriod=2
Jul 07 19:08:42.054009 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.053980    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"]
Jul 07 19:08:42.077869 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:42.077696    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"348357bd-0dd0-421f-bf5a-9ab237730434\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:42.109277 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.109253    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:42.109723 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.109690    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" containerName="manager"
Jul 07 19:08:42.109792 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.109727    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" containerName="manager"
Jul 07 19:08:42.109830 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.109804    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" containerName="manager"
Jul 07 19:08:42.112782 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.112764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.125655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.125635    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:42.174493 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:42.174447    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:42.209850 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.209821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.209967 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.209890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvcx\" (UniqueName: \"kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.278198 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.278169    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:42.278895 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.278876    2560 generic.go:356] "Generic (PLEG): container finished" podID="348357bd-0dd0-421f-bf5a-9ab237730434" containerID="ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0" exitCode=0
Jul 07 19:08:42.278954 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.278943    2560 scope.go:117] "RemoveContainer" containerID="ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0"
Jul 07 19:08:42.280089 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:42.280067    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:42.286525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.286508    2560 scope.go:117] "RemoveContainer" containerID="ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0"
Jul 07 19:08:42.286808 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:42.286782    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0\": container with ID starting with ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0 not found: ID does not exist" containerID="ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0"
Jul 07 19:08:42.286853 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.286818    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0"} err="failed to get container status \"ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0\": rpc error: code = NotFound desc = could not find container \"ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0\": container with ID starting with ab6aa4317ce48f111057c16e28b1880e7c0f864ddca4c9db97f3d7039878e8f0 not found: ID does not exist"
Jul 07 19:08:42.311208 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.311157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.311282 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.311216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvcx\" (UniqueName: \"kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.311503 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.311485    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.331742 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.331714    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvcx\" (UniqueName: \"kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hmmrn\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.412344 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.412321    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sln8c\" (UniqueName: \"kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c\") pod \"348357bd-0dd0-421f-bf5a-9ab237730434\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") "
Jul 07 19:08:42.412424 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.412354    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume\") pod \"348357bd-0dd0-421f-bf5a-9ab237730434\" (UID: \"348357bd-0dd0-421f-bf5a-9ab237730434\") "
Jul 07 19:08:42.412747 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.412725    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "348357bd-0dd0-421f-bf5a-9ab237730434" (UID: "348357bd-0dd0-421f-bf5a-9ab237730434"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:08:42.414271 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.414250    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c" (OuterVolumeSpecName: "kube-api-access-sln8c") pod "348357bd-0dd0-421f-bf5a-9ab237730434" (UID: "348357bd-0dd0-421f-bf5a-9ab237730434"). InnerVolumeSpecName "kube-api-access-sln8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:08:42.423410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.423390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:42.513373 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.513341    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sln8c\" (UniqueName: \"kubernetes.io/projected/348357bd-0dd0-421f-bf5a-9ab237730434-kube-api-access-sln8c\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:08:42.513483 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.513380    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/348357bd-0dd0-421f-bf5a-9ab237730434-extensions-socket-volume\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:08:42.563689 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:42.563663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:42.566039 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:08:42.566007    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode328c12c_c94e_45e6_b145_6f20f0ce64ae.slice/crio-ff417d71be474c2e3022c81c0584912032403c89f3c91d975e79f125ece9bbe0 WatchSource:0}: Error finding container ff417d71be474c2e3022c81c0584912032403c89f3c91d975e79f125ece9bbe0: Status 404 returned error can't find the container with id ff417d71be474c2e3022c81c0584912032403c89f3c91d975e79f125ece9bbe0
Jul 07 19:08:43.239930 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.239894    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:08:43.243275 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.243253    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.267255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.267228    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:08:43.289591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.289561    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:43.291432 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.291398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" event={"ID":"e328c12c-c94e-45e6-b145-6f20f0ce64ae","Type":"ContainerStarted","Data":"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"}
Jul 07 19:08:43.291534 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.291469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:43.291534 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.291485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" event={"ID":"e328c12c-c94e-45e6-b145-6f20f0ce64ae","Type":"ContainerStarted","Data":"ff417d71be474c2e3022c81c0584912032403c89f3c91d975e79f125ece9bbe0"}
Jul 07 19:08:43.298687 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:43.298645    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:43.319340 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.319319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.319437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.319369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcwm\" (UniqueName: \"kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.362513 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:43.362484    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:43.364999 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:43.364981    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6cnr9\" is forbidden: User \"system:node:ip-10-0-140-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-56.ec2.internal' and this object" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6cnr9"
Jul 07 19:08:43.407226 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.407202    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348357bd-0dd0-421f-bf5a-9ab237730434" path="/var/lib/kubelet/pods/348357bd-0dd0-421f-bf5a-9ab237730434/volumes"
Jul 07 19:08:43.419782 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.419763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.419892 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.419862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcwm\" (UniqueName: \"kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.420098 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.420080    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.435644 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.435621    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcwm\" (UniqueName: \"kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm\") pod \"kuadrant-operator-controller-manager-86878c769f-tk9j7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.552860 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.552840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:43.736486 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.736428    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" podStartSLOduration=1.7364087320000001 podStartE2EDuration="1.736408732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:43.358134054 +0000 UTC m=+489.077933244" watchObservedRunningTime="2026-07-07 19:08:43.736408732 +0000 UTC m=+489.456207922"
Jul 07 19:08:43.737062 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:43.736997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:08:43.741546 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:08:43.741517    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b578a3_af94_4446_b5e2_ac8d8bda60f7.slice/crio-110ea57b7d871109e2edab5aaefa3671baa7fbdeb6a56033e0d782f6a3d769fa WatchSource:0}: Error finding container 110ea57b7d871109e2edab5aaefa3671baa7fbdeb6a56033e0d782f6a3d769fa: Status 404 returned error can't find the container with id 110ea57b7d871109e2edab5aaefa3671baa7fbdeb6a56033e0d782f6a3d769fa
Jul 07 19:08:44.297283 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:44.297246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" event={"ID":"f8b578a3-af94-4446-b5e2-ac8d8bda60f7","Type":"ContainerStarted","Data":"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"}
Jul 07 19:08:44.297646 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:44.297306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" event={"ID":"f8b578a3-af94-4446-b5e2-ac8d8bda60f7","Type":"ContainerStarted","Data":"110ea57b7d871109e2edab5aaefa3671baa7fbdeb6a56033e0d782f6a3d769fa"}
Jul 07 19:08:44.297646 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:44.297413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:44.323398 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:44.323344    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" podStartSLOduration=1.323331848 podStartE2EDuration="1.323331848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:44.321646942 +0000 UTC m=+490.041446133" watchObservedRunningTime="2026-07-07 19:08:44.323331848 +0000 UTC m=+490.043131037"
Jul 07 19:08:54.299461 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:54.299427    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:55.304230 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.304200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:08:55.396512 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.396485    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:55.396737 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.396669    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" podUID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" containerName="manager" containerID="cri-o://bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add" gracePeriod=10
Jul 07 19:08:55.636646 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.636622    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:55.715531 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.715509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume\") pod \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") "
Jul 07 19:08:55.715640 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.715567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qvcx\" (UniqueName: \"kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx\") pod \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\" (UID: \"e328c12c-c94e-45e6-b145-6f20f0ce64ae\") "
Jul 07 19:08:55.715904 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.715879    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e328c12c-c94e-45e6-b145-6f20f0ce64ae" (UID: "e328c12c-c94e-45e6-b145-6f20f0ce64ae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:08:55.717461 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.717441    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx" (OuterVolumeSpecName: "kube-api-access-6qvcx") pod "e328c12c-c94e-45e6-b145-6f20f0ce64ae" (UID: "e328c12c-c94e-45e6-b145-6f20f0ce64ae"). InnerVolumeSpecName "kube-api-access-6qvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:08:55.816862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.816811    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e328c12c-c94e-45e6-b145-6f20f0ce64ae-extensions-socket-volume\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:08:55.816862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:55.816831    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvcx\" (UniqueName: \"kubernetes.io/projected/e328c12c-c94e-45e6-b145-6f20f0ce64ae-kube-api-access-6qvcx\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:08:56.349392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.349356    2560 generic.go:356] "Generic (PLEG): container finished" podID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" containerID="bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add" exitCode=0
Jul 07 19:08:56.349818 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.349399    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"
Jul 07 19:08:56.349818 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.349419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" event={"ID":"e328c12c-c94e-45e6-b145-6f20f0ce64ae","Type":"ContainerDied","Data":"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"}
Jul 07 19:08:56.349818 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.349460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn" event={"ID":"e328c12c-c94e-45e6-b145-6f20f0ce64ae","Type":"ContainerDied","Data":"ff417d71be474c2e3022c81c0584912032403c89f3c91d975e79f125ece9bbe0"}
Jul 07 19:08:56.349818 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.349482    2560 scope.go:117] "RemoveContainer" containerID="bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"
Jul 07 19:08:56.358002 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.357986    2560 scope.go:117] "RemoveContainer" containerID="bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"
Jul 07 19:08:56.358264 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:08:56.358242    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add\": container with ID starting with bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add not found: ID does not exist" containerID="bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"
Jul 07 19:08:56.358341 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.358271    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add"} err="failed to get container status \"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add\": rpc error: code = NotFound desc = could not find container \"bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add\": container with ID starting with bbb19b313c64a2780b7da739d0d4d0220ab1574bf724701276f4f1fa48417add not found: ID does not exist"
Jul 07 19:08:56.382908 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.382885    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:56.392931 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:56.392912    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hmmrn"]
Jul 07 19:08:57.407349 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:08:57.407313    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" path="/var/lib/kubelet/pods/e328c12c-c94e-45e6-b145-6f20f0ce64ae/volumes"
Jul 07 19:09:05.849483 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:05.849436    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:09:14.005831 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.005799    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:14.006202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.006077    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" containerName="manager"
Jul 07 19:09:14.006202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.006087    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" containerName="manager"
Jul 07 19:09:14.006202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.006135    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e328c12c-c94e-45e6-b145-6f20f0ce64ae" containerName="manager"
Jul 07 19:09:14.009241 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.009222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.013070 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.013044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 19:09:14.013327 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.013312    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f2sxg\""
Jul 07 19:09:14.021779 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.021760    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:14.096071 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.096039    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:14.144875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.144846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.144875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.144876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmn7\" (UniqueName: \"kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.245685 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.245656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.245829 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.245688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmn7\" (UniqueName: \"kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.246323 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.246305    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.265320 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.265264    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmn7\" (UniqueName: \"kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7\") pod \"limitador-limitador-64c94fd64f-9jqs9\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.318907 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.318881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:14.446042 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:14.445883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:14.448153 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:09:14.448119    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717c4b75_fc60_4947_98c2_9e6322154364.slice/crio-9f1a43477aa666d11b7820230fc7996f02a3e95a9be4ab7b43676e8644c9ba74 WatchSource:0}: Error finding container 9f1a43477aa666d11b7820230fc7996f02a3e95a9be4ab7b43676e8644c9ba74: Status 404 returned error can't find the container with id 9f1a43477aa666d11b7820230fc7996f02a3e95a9be4ab7b43676e8644c9ba74
Jul 07 19:09:15.427911 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:15.427878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" event={"ID":"717c4b75-fc60-4947-98c2-9e6322154364","Type":"ContainerStarted","Data":"9f1a43477aa666d11b7820230fc7996f02a3e95a9be4ab7b43676e8644c9ba74"}
Jul 07 19:09:17.436549 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:17.436515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" event={"ID":"717c4b75-fc60-4947-98c2-9e6322154364","Type":"ContainerStarted","Data":"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"}
Jul 07 19:09:17.436919 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:17.436583    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:17.471501 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:17.471455    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" podStartSLOduration=1.985263658 podStartE2EDuration="4.471442406s" totalImagesPullingTime="2.486178748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:17.470807102 +0000 UTC m=+523.190606292" watchObservedRunningTime="2026-07-07 19:09:17.471442406 +0000 UTC m=+523.191241597"
Jul 07 19:09:28.442390 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:28.442315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:30.870163 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:30.870124    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-89cc6879f-t7s8d" podUID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" containerName="console" containerID="cri-o://d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33" gracePeriod=15
Jul 07 19:09:31.113506 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.113481    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:09:31.175660 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.175660 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175632    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.175875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.175875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.175875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv8pn\" (UniqueName: \"kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.176031 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.176031 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.175923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config\") pod \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\" (UID: \"c0518f22-2776-46b8-b01c-03cf0bdcdc85\") "
Jul 07 19:09:31.176146 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.176079    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:31.176738 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.176673    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:31.176865 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.176773    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config" (OuterVolumeSpecName: "console-config") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:31.177044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.177021    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca" (OuterVolumeSpecName: "service-ca") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:31.177202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.177185    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-trusted-ca-bundle\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.177280 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.177208    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-oauth-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.177280 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.177222    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.178229 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.178196    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:09:31.178912 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.178871    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn" (OuterVolumeSpecName: "kube-api-access-jv8pn") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "kube-api-access-jv8pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:09:31.181021 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.180995    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c0518f22-2776-46b8-b01c-03cf0bdcdc85" (UID: "c0518f22-2776-46b8-b01c-03cf0bdcdc85"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:09:31.277827 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.277806    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv8pn\" (UniqueName: \"kubernetes.io/projected/c0518f22-2776-46b8-b01c-03cf0bdcdc85-kube-api-access-jv8pn\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.277917 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.277829    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0518f22-2776-46b8-b01c-03cf0bdcdc85-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.277917 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.277839    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-serving-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.277917 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.277847    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0518f22-2776-46b8-b01c-03cf0bdcdc85-console-oauth-config\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:31.493020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.492964    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" containerID="d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33" exitCode=0
Jul 07 19:09:31.493127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.493036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89cc6879f-t7s8d"
Jul 07 19:09:31.493127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.493068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89cc6879f-t7s8d" event={"ID":"c0518f22-2776-46b8-b01c-03cf0bdcdc85","Type":"ContainerDied","Data":"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"}
Jul 07 19:09:31.493127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.493124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89cc6879f-t7s8d" event={"ID":"c0518f22-2776-46b8-b01c-03cf0bdcdc85","Type":"ContainerDied","Data":"396bdffcc61a3486a8937f6c49e3236f4910882b10f908f9fa19dd1c372f386b"}
Jul 07 19:09:31.493285 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.493152    2560 scope.go:117] "RemoveContainer" containerID="d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"
Jul 07 19:09:31.500954 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.500938    2560 scope.go:117] "RemoveContainer" containerID="d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"
Jul 07 19:09:31.501209 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:09:31.501191    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33\": container with ID starting with d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33 not found: ID does not exist" containerID="d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"
Jul 07 19:09:31.501252 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.501217    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33"} err="failed to get container status \"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33\": rpc error: code = NotFound desc = could not find container \"d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33\": container with ID starting with d94da95c95e7fe122ee273313bf8b06a7197463c9326a1befea0a33c6d123a33 not found: ID does not exist"
Jul 07 19:09:31.511470 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.511446    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:09:31.516474 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:31.516454    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-89cc6879f-t7s8d"]
Jul 07 19:09:32.937599 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:32.937566    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:32.938072 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:32.937804    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" podUID="717c4b75-fc60-4947-98c2-9e6322154364" containerName="limitador" containerID="cri-o://2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7" gracePeriod=30
Jul 07 19:09:33.409795 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.409764    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" path="/var/lib/kubelet/pods/c0518f22-2776-46b8-b01c-03cf0bdcdc85/volumes"
Jul 07 19:09:33.481585 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.481566    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:33.492574 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.492555    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file\") pod \"717c4b75-fc60-4947-98c2-9e6322154364\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") "
Jul 07 19:09:33.492642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.492584    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtmn7\" (UniqueName: \"kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7\") pod \"717c4b75-fc60-4947-98c2-9e6322154364\" (UID: \"717c4b75-fc60-4947-98c2-9e6322154364\") "
Jul 07 19:09:33.492927 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.492899    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file" (OuterVolumeSpecName: "config-file") pod "717c4b75-fc60-4947-98c2-9e6322154364" (UID: "717c4b75-fc60-4947-98c2-9e6322154364"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:33.494521 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.494499    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7" (OuterVolumeSpecName: "kube-api-access-gtmn7") pod "717c4b75-fc60-4947-98c2-9e6322154364" (UID: "717c4b75-fc60-4947-98c2-9e6322154364"). InnerVolumeSpecName "kube-api-access-gtmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:09:33.505181 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.505155    2560 generic.go:356] "Generic (PLEG): container finished" podID="717c4b75-fc60-4947-98c2-9e6322154364" containerID="2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7" exitCode=0
Jul 07 19:09:33.505271 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.505197    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"
Jul 07 19:09:33.505271 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.505238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" event={"ID":"717c4b75-fc60-4947-98c2-9e6322154364","Type":"ContainerDied","Data":"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"}
Jul 07 19:09:33.505340 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.505286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9jqs9" event={"ID":"717c4b75-fc60-4947-98c2-9e6322154364","Type":"ContainerDied","Data":"9f1a43477aa666d11b7820230fc7996f02a3e95a9be4ab7b43676e8644c9ba74"}
Jul 07 19:09:33.505340 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.505307    2560 scope.go:117] "RemoveContainer" containerID="2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"
Jul 07 19:09:33.513919 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.513903    2560 scope.go:117] "RemoveContainer" containerID="2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"
Jul 07 19:09:33.514175 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:09:33.514159    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7\": container with ID starting with 2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7 not found: ID does not exist" containerID="2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"
Jul 07 19:09:33.514249 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.514195    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7"} err="failed to get container status \"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7\": rpc error: code = NotFound desc = could not find container \"2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7\": container with ID starting with 2b9556d69d8051be89766ef0a03e576fe6077249bb77b0cbd1c2dd993e9325b7 not found: ID does not exist"
Jul 07 19:09:33.526497 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.526476    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:33.529249 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.529226    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9jqs9"]
Jul 07 19:09:33.594019 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.593995    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717c4b75-fc60-4947-98c2-9e6322154364-config-file\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:33.594019 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:33.594016    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmn7\" (UniqueName: \"kubernetes.io/projected/717c4b75-fc60-4947-98c2-9e6322154364-kube-api-access-gtmn7\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:09:35.406957 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.406925    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717c4b75-fc60-4947-98c2-9e6322154364" path="/var/lib/kubelet/pods/717c4b75-fc60-4947-98c2-9e6322154364/volumes"
Jul 07 19:09:35.490228 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490171    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-vp45s"]
Jul 07 19:09:35.490445 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490433    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" containerName="console"
Jul 07 19:09:35.490494 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490446    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" containerName="console"
Jul 07 19:09:35.490494 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490462    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717c4b75-fc60-4947-98c2-9e6322154364" containerName="limitador"
Jul 07 19:09:35.490494 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490468    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="717c4b75-fc60-4947-98c2-9e6322154364" containerName="limitador"
Jul 07 19:09:35.490586 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490516    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0518f22-2776-46b8-b01c-03cf0bdcdc85" containerName="console"
Jul 07 19:09:35.490586 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.490523    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="717c4b75-fc60-4947-98c2-9e6322154364" containerName="limitador"
Jul 07 19:09:35.494496 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.494477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.496489 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.496465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 07 19:09:35.496591 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.496558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jf6qq\""
Jul 07 19:09:35.502566 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.502542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47047fc2-fd87-454a-b30f-be0583f1767c-data\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.502685 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.502593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5npc\" (UniqueName: \"kubernetes.io/projected/47047fc2-fd87-454a-b30f-be0583f1767c-kube-api-access-m5npc\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.503525 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.503505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-vp45s"]
Jul 07 19:09:35.603200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.603172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47047fc2-fd87-454a-b30f-be0583f1767c-data\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.603347 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.603213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5npc\" (UniqueName: \"kubernetes.io/projected/47047fc2-fd87-454a-b30f-be0583f1767c-kube-api-access-m5npc\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.603516 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.603498    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47047fc2-fd87-454a-b30f-be0583f1767c-data\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.613006 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.612981    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5npc\" (UniqueName: \"kubernetes.io/projected/47047fc2-fd87-454a-b30f-be0583f1767c-kube-api-access-m5npc\") pod \"postgres-69fb4594cf-vp45s\" (UID: \"47047fc2-fd87-454a-b30f-be0583f1767c\") " pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.805933 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.805910    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:35.932263 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:35.932240    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-vp45s"]
Jul 07 19:09:35.934827 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:09:35.934801    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47047fc2_fd87_454a_b30f_be0583f1767c.slice/crio-ecb4782dfabf8d0c50dc71205589c0c1272b28917bfbb5c8b682abfa980f31ff WatchSource:0}: Error finding container ecb4782dfabf8d0c50dc71205589c0c1272b28917bfbb5c8b682abfa980f31ff: Status 404 returned error can't find the container with id ecb4782dfabf8d0c50dc71205589c0c1272b28917bfbb5c8b682abfa980f31ff
Jul 07 19:09:36.522044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:36.521994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-vp45s" event={"ID":"47047fc2-fd87-454a-b30f-be0583f1767c","Type":"ContainerStarted","Data":"ecb4782dfabf8d0c50dc71205589c0c1272b28917bfbb5c8b682abfa980f31ff"}
Jul 07 19:09:40.540962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:40.540930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-vp45s" event={"ID":"47047fc2-fd87-454a-b30f-be0583f1767c","Type":"ContainerStarted","Data":"fa2f26b057654e01b014aa00bc9e9335e7fedb52a35bffeb80bf23b48b9d8617"}
Jul 07 19:09:40.541338 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:40.541041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:40.561920 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:40.561880    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-vp45s" podStartSLOduration=1.124261227 podStartE2EDuration="5.561866634s" totalImagesPullingTime="4.437605407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:40.560592827 +0000 UTC m=+546.280392028" watchObservedRunningTime="2026-07-07 19:09:40.561866634 +0000 UTC m=+546.281665823"
Jul 07 19:09:46.573985 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:46.573959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-vp45s"
Jul 07 19:09:53.661316 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.661283    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:09:53.680909 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.680884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.683712 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.683679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 19:09:53.683852 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.683676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 19:09:53.683852 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.683830    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pvllf\""
Jul 07 19:09:53.686280 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.686257    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:09:53.733198 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.733175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbg8m\" (UniqueName: \"kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.733320 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.733222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.833921 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.833895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbg8m\" (UniqueName: \"kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.834020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.833940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.836192 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.836174    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.848801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.848782    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbg8m\" (UniqueName: \"kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m\") pod \"maas-controller-7f59b8955f-pdgvx\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") " pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:53.914747 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.914669    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:09:53.917323 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.917308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:53.934831 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.934811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:09:53.992667 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:53.992644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:54.035604 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.035578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.035779 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.035625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxs8\" (UniqueName: \"kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.119002 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.118978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:09:54.121317 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:09:54.121291    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78901eb_3d72_4f06_a172_b5271341b9c5.slice/crio-ec775d33ab0195ee99885a3de87bb5368f25f03a9d5ffd6e5f72c49d38021133 WatchSource:0}: Error finding container ec775d33ab0195ee99885a3de87bb5368f25f03a9d5ffd6e5f72c49d38021133: Status 404 returned error can't find the container with id ec775d33ab0195ee99885a3de87bb5368f25f03a9d5ffd6e5f72c49d38021133
Jul 07 19:09:54.136503 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.136483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.136592 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.136520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxs8\" (UniqueName: \"kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.138673 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.138650    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.145682 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.145664    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxs8\" (UniqueName: \"kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8\") pod \"maas-controller-57c89485cc-fpbkv\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") " pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.227716 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.227653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:54.352304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.352274    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:09:54.355233 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:09:54.355208    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbbfd9a_7951_4602_a47c_0bd1eef6bed7.slice/crio-4a3642877465d14c89838e3f18451f65b5458d453134f0021b39842a60034f54 WatchSource:0}: Error finding container 4a3642877465d14c89838e3f18451f65b5458d453134f0021b39842a60034f54: Status 404 returned error can't find the container with id 4a3642877465d14c89838e3f18451f65b5458d453134f0021b39842a60034f54
Jul 07 19:09:54.597109 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.597074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" event={"ID":"c78901eb-3d72-4f06-a172-b5271341b9c5","Type":"ContainerStarted","Data":"ec775d33ab0195ee99885a3de87bb5368f25f03a9d5ffd6e5f72c49d38021133"}
Jul 07 19:09:54.598538 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:54.598512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-fpbkv" event={"ID":"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7","Type":"ContainerStarted","Data":"4a3642877465d14c89838e3f18451f65b5458d453134f0021b39842a60034f54"}
Jul 07 19:09:56.609673 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.609575    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" event={"ID":"c78901eb-3d72-4f06-a172-b5271341b9c5","Type":"ContainerStarted","Data":"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"}
Jul 07 19:09:56.610083 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.609742    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:09:56.611090 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.611063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-fpbkv" event={"ID":"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7","Type":"ContainerStarted","Data":"c994a02b944203929ea6ab7c398fb6e7d40eaa891ad974296f66f6036940f3af"}
Jul 07 19:09:56.611218 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.611175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:09:56.631924 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.631884    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" podStartSLOduration=1.481475514 podStartE2EDuration="3.631869767s" totalImagesPullingTime="2.150394253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:56.630810166 +0000 UTC m=+562.350609355" watchObservedRunningTime="2026-07-07 19:09:56.631869767 +0000 UTC m=+562.351668956"
Jul 07 19:09:56.661962 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:09:56.661927    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57c89485cc-fpbkv" podStartSLOduration=1.73590713 podStartE2EDuration="3.661914337s" totalImagesPullingTime="1.926007207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:56.660489488 +0000 UTC m=+562.380288678" watchObservedRunningTime="2026-07-07 19:09:56.661914337 +0000 UTC m=+562.381713526"
Jul 07 19:10:07.623062 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:07.623031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:10:07.623492 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:07.623143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:10:07.715429 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:07.715392    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:10:07.715658 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:07.715633    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" podUID="c78901eb-3d72-4f06-a172-b5271341b9c5" containerName="manager" containerID="cri-o://1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe" gracePeriod=30
Jul 07 19:10:07.959530 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:07.959508    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:10:08.036467 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.036434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbg8m\" (UniqueName: \"kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m\") pod \"c78901eb-3d72-4f06-a172-b5271341b9c5\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") "
Jul 07 19:10:08.036575 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.036473    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert\") pod \"c78901eb-3d72-4f06-a172-b5271341b9c5\" (UID: \"c78901eb-3d72-4f06-a172-b5271341b9c5\") "
Jul 07 19:10:08.038477 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.038454    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c78901eb-3d72-4f06-a172-b5271341b9c5" (UID: "c78901eb-3d72-4f06-a172-b5271341b9c5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:10:08.038537 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.038521    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m" (OuterVolumeSpecName: "kube-api-access-jbg8m") pod "c78901eb-3d72-4f06-a172-b5271341b9c5" (UID: "c78901eb-3d72-4f06-a172-b5271341b9c5"). InnerVolumeSpecName "kube-api-access-jbg8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:10:08.137094 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.137043    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbg8m\" (UniqueName: \"kubernetes.io/projected/c78901eb-3d72-4f06-a172-b5271341b9c5-kube-api-access-jbg8m\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:10:08.137094 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.137064    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c78901eb-3d72-4f06-a172-b5271341b9c5-webhook-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:10:08.663626 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.663594    2560 generic.go:356] "Generic (PLEG): container finished" podID="c78901eb-3d72-4f06-a172-b5271341b9c5" containerID="1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe" exitCode=0
Jul 07 19:10:08.664035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.663639    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f59b8955f-pdgvx"
Jul 07 19:10:08.664035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.663681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" event={"ID":"c78901eb-3d72-4f06-a172-b5271341b9c5","Type":"ContainerDied","Data":"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"}
Jul 07 19:10:08.664035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.663751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f59b8955f-pdgvx" event={"ID":"c78901eb-3d72-4f06-a172-b5271341b9c5","Type":"ContainerDied","Data":"ec775d33ab0195ee99885a3de87bb5368f25f03a9d5ffd6e5f72c49d38021133"}
Jul 07 19:10:08.664035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.663774    2560 scope.go:117] "RemoveContainer" containerID="1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"
Jul 07 19:10:08.672436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.672418    2560 scope.go:117] "RemoveContainer" containerID="1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"
Jul 07 19:10:08.672682 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:10:08.672663    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe\": container with ID starting with 1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe not found: ID does not exist" containerID="1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"
Jul 07 19:10:08.672773 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.672691    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe"} err="failed to get container status \"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe\": rpc error: code = NotFound desc = could not find container \"1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe\": container with ID starting with 1c08cbc7ce5055f7a7ad5cae7ff0ef8aabf8580d0e29a41fb4156c6d8bae62fe not found: ID does not exist"
Jul 07 19:10:08.688091 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.688068    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:10:08.692462 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:08.692444    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f59b8955f-pdgvx"]
Jul 07 19:10:09.407299 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:09.407265    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78901eb-3d72-4f06-a172-b5271341b9c5" path="/var/lib/kubelet/pods/c78901eb-3d72-4f06-a172-b5271341b9c5/volumes"
Jul 07 19:10:32.157447 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.157416    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-867c89778c-ct94l"]
Jul 07 19:10:32.157834 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.157710    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78901eb-3d72-4f06-a172-b5271341b9c5" containerName="manager"
Jul 07 19:10:32.157834 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.157723    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78901eb-3d72-4f06-a172-b5271341b9c5" containerName="manager"
Jul 07 19:10:32.157834 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.157786    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78901eb-3d72-4f06-a172-b5271341b9c5" containerName="manager"
Jul 07 19:10:32.160355 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.160334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.162335 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.162312    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 07 19:10:32.162437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.162316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-xkgbb\""
Jul 07 19:10:32.175747 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.175721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-867c89778c-ct94l"]
Jul 07 19:10:32.315054 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.315022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d9w\" (UniqueName: \"kubernetes.io/projected/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-kube-api-access-f9d9w\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.315202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.315061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-openshift-service-ca\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.315202 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.315086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-maas-api-tls\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.416253 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.416159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-openshift-service-ca\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.416253 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.416217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-maas-api-tls\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.416451 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.416300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d9w\" (UniqueName: \"kubernetes.io/projected/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-kube-api-access-f9d9w\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.416996 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.416959    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-openshift-service-ca\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.418950 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.418922    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-maas-api-tls\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.424870 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.424847    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d9w\" (UniqueName: \"kubernetes.io/projected/2645e17e-7e3f-4d3d-a191-66cadbeb4b4f-kube-api-access-f9d9w\") pod \"maas-api-867c89778c-ct94l\" (UID: \"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f\") " pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.470811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.470782    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:32.612646 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.612600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-867c89778c-ct94l"]
Jul 07 19:10:32.615190 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:10:32.615163    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2645e17e_7e3f_4d3d_a191_66cadbeb4b4f.slice/crio-3a2da551cc8fb46e481c132f6ff5c8a22cd8a2eb0f3fffccdfcb027d45a00a49 WatchSource:0}: Error finding container 3a2da551cc8fb46e481c132f6ff5c8a22cd8a2eb0f3fffccdfcb027d45a00a49: Status 404 returned error can't find the container with id 3a2da551cc8fb46e481c132f6ff5c8a22cd8a2eb0f3fffccdfcb027d45a00a49
Jul 07 19:10:32.757544 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.757516    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-bjtzz"]
Jul 07 19:10:32.760891 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.760869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:32.762894 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.762875    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 19:10:32.762988 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.762905    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mn7ct\""
Jul 07 19:10:32.764183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.764159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-867c89778c-ct94l" event={"ID":"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f","Type":"ContainerStarted","Data":"3a2da551cc8fb46e481c132f6ff5c8a22cd8a2eb0f3fffccdfcb027d45a00a49"}
Jul 07 19:10:32.770258 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.770233    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-bjtzz"]
Jul 07 19:10:32.818836 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.818764    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-wgd92"]
Jul 07 19:10:32.820933 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.820917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:32.830744 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.830721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-wgd92"]
Jul 07 19:10:32.921370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.921334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5be2b7-e91f-4595-9631-01e25e1c81a5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:32.921626 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.921596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbgql\" (UniqueName: \"kubernetes.io/projected/de5be2b7-e91f-4595-9631-01e25e1c81a5-kube-api-access-lbgql\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:32.921778 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.921672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-plugins-config-volume\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:32.921778 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:32.921759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdc2g\" (UniqueName: \"kubernetes.io/projected/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-kube-api-access-kdc2g\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.023039 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.022964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-plugins-config-volume\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.023039 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.023007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdc2g\" (UniqueName: \"kubernetes.io/projected/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-kube-api-access-kdc2g\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.023230 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.023047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5be2b7-e91f-4595-9631-01e25e1c81a5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:33.023230 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.023072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgql\" (UniqueName: \"kubernetes.io/projected/de5be2b7-e91f-4595-9631-01e25e1c81a5-kube-api-access-lbgql\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:33.023764 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.023740    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-plugins-config-volume\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.023866 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.023793    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5be2b7-e91f-4595-9631-01e25e1c81a5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:33.034158 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.034137    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbgql\" (UniqueName: \"kubernetes.io/projected/de5be2b7-e91f-4595-9631-01e25e1c81a5-kube-api-access-lbgql\") pod \"payload-pre-processing-7964457696-wgd92\" (UID: \"de5be2b7-e91f-4595-9631-01e25e1c81a5\") " pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:33.034228 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.034186    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdc2g\" (UniqueName: \"kubernetes.io/projected/6d4ba065-b74b-49c9-b3d8-7e58b070e7d2-kube-api-access-kdc2g\") pod \"payload-processing-58f86c7798-bjtzz\" (UID: \"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2\") " pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.073624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.073599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:33.130945 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.130918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:33.204234 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.204117    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-bjtzz"]
Jul 07 19:10:33.206887 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:10:33.206854    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4ba065_b74b_49c9_b3d8_7e58b070e7d2.slice/crio-d437a3c438fcb46182714ebdbd2e8e2fe5e3b7de67946ca843ceb91c2d016bc7 WatchSource:0}: Error finding container d437a3c438fcb46182714ebdbd2e8e2fe5e3b7de67946ca843ceb91c2d016bc7: Status 404 returned error can't find the container with id d437a3c438fcb46182714ebdbd2e8e2fe5e3b7de67946ca843ceb91c2d016bc7
Jul 07 19:10:33.262285 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.262156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-wgd92"]
Jul 07 19:10:33.265263 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:10:33.265229    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5be2b7_e91f_4595_9631_01e25e1c81a5.slice/crio-ac7b171207d2e7f1050ed7999c10956a0f10e9e7b27973cd0e0d8d1d6b77708c WatchSource:0}: Error finding container ac7b171207d2e7f1050ed7999c10956a0f10e9e7b27973cd0e0d8d1d6b77708c: Status 404 returned error can't find the container with id ac7b171207d2e7f1050ed7999c10956a0f10e9e7b27973cd0e0d8d1d6b77708c
Jul 07 19:10:33.772335 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.772213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92" event={"ID":"de5be2b7-e91f-4595-9631-01e25e1c81a5","Type":"ContainerStarted","Data":"ac7b171207d2e7f1050ed7999c10956a0f10e9e7b27973cd0e0d8d1d6b77708c"}
Jul 07 19:10:33.773588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:33.773529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz" event={"ID":"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2","Type":"ContainerStarted","Data":"d437a3c438fcb46182714ebdbd2e8e2fe5e3b7de67946ca843ceb91c2d016bc7"}
Jul 07 19:10:34.779434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.779405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92" event={"ID":"de5be2b7-e91f-4595-9631-01e25e1c81a5","Type":"ContainerStarted","Data":"ebefc6135be60f1f7b9646e391b30a02fda6a4e99dc17d57d14987552f26184e"}
Jul 07 19:10:34.779805 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.779516    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:10:34.781243 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.781218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz" event={"ID":"6d4ba065-b74b-49c9-b3d8-7e58b070e7d2","Type":"ContainerStarted","Data":"f54ed44b1364ea0d8097b86d9f4be1e454e424d02544360ab7405d5531fea1f7"}
Jul 07 19:10:34.781336 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.781260    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:34.782533 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.782515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-867c89778c-ct94l" event={"ID":"2645e17e-7e3f-4d3d-a191-66cadbeb4b4f","Type":"ContainerStarted","Data":"3ea28ac12c8af61f525f1502e8a9f638ae01a913d5fa2b440726a3a5a41337e9"}
Jul 07 19:10:34.782662 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.782648    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:34.800755 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.799582    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92" podStartSLOduration=1.394249943 podStartE2EDuration="2.799563711s" totalImagesPullingTime="1.405313768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:34.798324356 +0000 UTC m=+600.518123546" watchObservedRunningTime="2026-07-07 19:10:34.799563711 +0000 UTC m=+600.519362905"
Jul 07 19:10:34.834000 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.833953    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-867c89778c-ct94l" podStartSLOduration=0.778944987 podStartE2EDuration="2.833933926s" totalImagesPullingTime="2.054988939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:34.832778535 +0000 UTC m=+600.552577728" watchObservedRunningTime="2026-07-07 19:10:34.833933926 +0000 UTC m=+600.553733117"
Jul 07 19:10:34.834398 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:34.834378    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz" podStartSLOduration=1.371795237 podStartE2EDuration="2.834370522s" totalImagesPullingTime="1.462575285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:34.815933531 +0000 UTC m=+600.535732723" watchObservedRunningTime="2026-07-07 19:10:34.834370522 +0000 UTC m=+600.554169711"
Jul 07 19:10:40.796620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:40.796591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-867c89778c-ct94l"
Jul 07 19:10:45.787867 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:45.787837    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-bjtzz"
Jul 07 19:10:45.788294 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:10:45.787918    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-wgd92"
Jul 07 19:11:14.457119 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.457075    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"]
Jul 07 19:11:14.459413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.459395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.462200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.462154    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 07 19:11:14.462200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.462165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 19:11:14.462200 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.462187    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 19:11:14.462413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.462165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n4slz\""
Jul 07 19:11:14.471625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.471604    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"]
Jul 07 19:11:14.532495 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.532642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.532642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.532642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.532642 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.532837 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.532645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn42v\" (UniqueName: \"kubernetes.io/projected/6c33c50c-2ad0-405d-beba-245bc30a8ed9-kube-api-access-bn42v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633038 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633038 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn42v\" (UniqueName: \"kubernetes.io/projected/6c33c50c-2ad0-405d-beba-245bc30a8ed9-kube-api-access-bn42v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633518 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633475    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633518 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633503    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.633727 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.633694    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.635457 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.635435    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c33c50c-2ad0-405d-beba-245bc30a8ed9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.635659 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.635644    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c33c50c-2ad0-405d-beba-245bc30a8ed9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.643529 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.643509    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn42v\" (UniqueName: \"kubernetes.io/projected/6c33c50c-2ad0-405d-beba-245bc30a8ed9-kube-api-access-bn42v\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tswks\" (UID: \"6c33c50c-2ad0-405d-beba-245bc30a8ed9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.771974 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.771947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:14.907447 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.907421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"]
Jul 07 19:11:14.910623 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:11:14.910595    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c33c50c_2ad0_405d_beba_245bc30a8ed9.slice/crio-10c6c6898d9d180112a1e47e4ed37d875714864cad58f51203bb81b4e1404ecc WatchSource:0}: Error finding container 10c6c6898d9d180112a1e47e4ed37d875714864cad58f51203bb81b4e1404ecc: Status 404 returned error can't find the container with id 10c6c6898d9d180112a1e47e4ed37d875714864cad58f51203bb81b4e1404ecc
Jul 07 19:11:14.911897 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.911879    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:11:14.952763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:14.952724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks" event={"ID":"6c33c50c-2ad0-405d-beba-245bc30a8ed9","Type":"ContainerStarted","Data":"10c6c6898d9d180112a1e47e4ed37d875714864cad58f51203bb81b4e1404ecc"}
Jul 07 19:11:16.963618 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:16.963586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks" event={"ID":"6c33c50c-2ad0-405d-beba-245bc30a8ed9","Type":"ContainerStarted","Data":"796c243c6b3b4bd98c91e5a056e5f8cec188f5ffa49b0f7913caa0fd18b86f68"}
Jul 07 19:11:16.988694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:16.988636    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks" podStartSLOduration=1.888955757 podStartE2EDuration="2.988612295s" totalImagesPullingTime="1.099656538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:16.986857674 +0000 UTC m=+642.706656865" watchObservedRunningTime="2026-07-07 19:11:16.988612295 +0000 UTC m=+642.708411487"
Jul 07 19:11:24.773004 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:24.772970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:24.773410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:24.773053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:24.785411 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:24.785391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:25.007424 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:25.007401    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tswks"
Jul 07 19:11:26.660748 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.660695    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"]
Jul 07 19:11:26.663609 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.663591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.665766 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.665748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 07 19:11:26.675910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.675888    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"]
Jul 07 19:11:26.723125 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrgb8\" (UniqueName: \"kubernetes.io/projected/fcfaf158-1bdb-4cd0-84ab-ec0202155685-kube-api-access-zrgb8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.723227 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.723227 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.723319 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.723354 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.723413 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.723398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824683 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824972 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.824972 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.824898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgb8\" (UniqueName: \"kubernetes.io/projected/fcfaf158-1bdb-4cd0-84ab-ec0202155685-kube-api-access-zrgb8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.825136 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.825051    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.825877 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.825375    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.825877 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.825529    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.827669 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.827647    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcfaf158-1bdb-4cd0-84ab-ec0202155685-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.828035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.828017    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfaf158-1bdb-4cd0-84ab-ec0202155685-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.835244 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.835220    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgb8\" (UniqueName: \"kubernetes.io/projected/fcfaf158-1bdb-4cd0-84ab-ec0202155685-kube-api-access-zrgb8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trfx9\" (UID: \"fcfaf158-1bdb-4cd0-84ab-ec0202155685\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:26.974940 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:26.974872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:27.106394 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:27.106361    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"]
Jul 07 19:11:27.110223 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:11:27.110191    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfaf158_1bdb_4cd0_84ab_ec0202155685.slice/crio-bb90ada48ceaebcad38b271b18a9cba40dcb8d4bb7b205dd722bc163c1e3718d WatchSource:0}: Error finding container bb90ada48ceaebcad38b271b18a9cba40dcb8d4bb7b205dd722bc163c1e3718d: Status 404 returned error can't find the container with id bb90ada48ceaebcad38b271b18a9cba40dcb8d4bb7b205dd722bc163c1e3718d
Jul 07 19:11:28.010279 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:28.010241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9" event={"ID":"fcfaf158-1bdb-4cd0-84ab-ec0202155685","Type":"ContainerStarted","Data":"0b75b2056250a2d69285ab9151557070820ac2e3d0cc61b46820c1d8f21833e3"}
Jul 07 19:11:28.010750 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:28.010320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9" event={"ID":"fcfaf158-1bdb-4cd0-84ab-ec0202155685","Type":"ContainerStarted","Data":"bb90ada48ceaebcad38b271b18a9cba40dcb8d4bb7b205dd722bc163c1e3718d"}
Jul 07 19:11:28.034592 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:28.034543    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9" podStartSLOduration=1.846844143 podStartE2EDuration="2.034529866s" totalImagesPullingTime="187.685723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:28.033458475 +0000 UTC m=+653.753257665" watchObservedRunningTime="2026-07-07 19:11:28.034529866 +0000 UTC m=+653.754329055"
Jul 07 19:11:36.975771 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:36.975740    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:36.976242 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:36.975807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:36.988082 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:36.988060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:37.060476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:37.060448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trfx9"
Jul 07 19:11:43.161432 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.161398    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"]
Jul 07 19:11:43.164198 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.164181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.166214 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.166185    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 07 19:11:43.176624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.176597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"]
Jul 07 19:11:43.252157 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.252302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.252302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.252302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzhp\" (UniqueName: \"kubernetes.io/projected/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-kube-api-access-qqzhp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.252302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.252302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.252278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353094 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353094 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353300 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzhp\" (UniqueName: \"kubernetes.io/projected/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-kube-api-access-qqzhp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353300 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353300 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353300 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353669 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353636    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353821 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353680    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.353821 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.353732    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.355478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.355455    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.355560 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.355520    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.362940 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.362918    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzhp\" (UniqueName: \"kubernetes.io/projected/67ad0a1b-e3d9-4ca9-b8db-544e9a159316-kube-api-access-qqzhp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hhjq6\" (UID: \"67ad0a1b-e3d9-4ca9-b8db-544e9a159316\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.478947 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.478884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:43.613833 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:43.613811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"]
Jul 07 19:11:43.615598 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:11:43.615576    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad0a1b_e3d9_4ca9_b8db_544e9a159316.slice/crio-70ef0236d7f77fe923a5cd33da2931c02f62887b7b31c94f66a39da01f0b1a48 WatchSource:0}: Error finding container 70ef0236d7f77fe923a5cd33da2931c02f62887b7b31c94f66a39da01f0b1a48: Status 404 returned error can't find the container with id 70ef0236d7f77fe923a5cd33da2931c02f62887b7b31c94f66a39da01f0b1a48
Jul 07 19:11:44.080351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:44.080274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6" event={"ID":"67ad0a1b-e3d9-4ca9-b8db-544e9a159316","Type":"ContainerStarted","Data":"6e08d4434396c8693e53abd88418820d58caf88c2e53c475a37226bbcbb2befa"}
Jul 07 19:11:44.080351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:44.080324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6" event={"ID":"67ad0a1b-e3d9-4ca9-b8db-544e9a159316","Type":"ContainerStarted","Data":"70ef0236d7f77fe923a5cd33da2931c02f62887b7b31c94f66a39da01f0b1a48"}
Jul 07 19:11:44.103637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:44.103590    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6" podStartSLOduration=0.893463214 podStartE2EDuration="1.103575724s" totalImagesPullingTime="210.11251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:44.102634404 +0000 UTC m=+669.822433594" watchObservedRunningTime="2026-07-07 19:11:44.103575724 +0000 UTC m=+669.823374914"
Jul 07 19:11:53.479342 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:53.479305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:53.479809 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:53.479360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:53.494351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:53.494325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:11:54.136511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:11:54.136482    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hhjq6"
Jul 07 19:14:02.532069 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.531978    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:14:02.532683 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.532193    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-57c89485cc-fpbkv" podUID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" containerName="manager" containerID="cri-o://c994a02b944203929ea6ab7c398fb6e7d40eaa891ad974296f66f6036940f3af" gracePeriod=30
Jul 07 19:14:02.703740 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.703686    2560 generic.go:356] "Generic (PLEG): container finished" podID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" containerID="c994a02b944203929ea6ab7c398fb6e7d40eaa891ad974296f66f6036940f3af" exitCode=0
Jul 07 19:14:02.703883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.703734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-fpbkv" event={"ID":"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7","Type":"ContainerDied","Data":"c994a02b944203929ea6ab7c398fb6e7d40eaa891ad974296f66f6036940f3af"}
Jul 07 19:14:02.781412 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.781385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:14:02.916524 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.916446    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwxs8\" (UniqueName: \"kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8\") pod \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") "
Jul 07 19:14:02.916524 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.916512    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert\") pod \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\" (UID: \"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7\") "
Jul 07 19:14:02.918603 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.918579    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8" (OuterVolumeSpecName: "kube-api-access-pwxs8") pod "2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" (UID: "2fbbfd9a-7951-4602-a47c-0bd1eef6bed7"). InnerVolumeSpecName "kube-api-access-pwxs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:14:02.918734 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:02.918598    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" (UID: "2fbbfd9a-7951-4602-a47c-0bd1eef6bed7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:14:03.017635 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.017607    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxs8\" (UniqueName: \"kubernetes.io/projected/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-kube-api-access-pwxs8\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:14:03.017635 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.017631    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7-webhook-cert\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:14:03.709001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.708927    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-fpbkv"
Jul 07 19:14:03.709423 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.708929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-fpbkv" event={"ID":"2fbbfd9a-7951-4602-a47c-0bd1eef6bed7","Type":"ContainerDied","Data":"4a3642877465d14c89838e3f18451f65b5458d453134f0021b39842a60034f54"}
Jul 07 19:14:03.709423 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.709053    2560 scope.go:117] "RemoveContainer" containerID="c994a02b944203929ea6ab7c398fb6e7d40eaa891ad974296f66f6036940f3af"
Jul 07 19:14:03.726377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.726354    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:14:03.730092 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:03.730066    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57c89485cc-fpbkv"]
Jul 07 19:14:04.521146 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.521114    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57c89485cc-pfqzg"]
Jul 07 19:14:04.521463 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.521451    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" containerName="manager"
Jul 07 19:14:04.521510 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.521465    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" containerName="manager"
Jul 07 19:14:04.521548 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.521538    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" containerName="manager"
Jul 07 19:14:04.524235 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.524219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.527402 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.527378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 19:14:04.527402 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.527394    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 19:14:04.527599 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.527378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pvllf\""
Jul 07 19:14:04.537008 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.536988    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c89485cc-pfqzg"]
Jul 07 19:14:04.630636 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.630615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgb5s\" (UniqueName: \"kubernetes.io/projected/3adf443a-f972-4340-980e-a52ae46ca00a-kube-api-access-zgb5s\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.630756 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.630658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3adf443a-f972-4340-980e-a52ae46ca00a-webhook-cert\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.731535 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.731508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgb5s\" (UniqueName: \"kubernetes.io/projected/3adf443a-f972-4340-980e-a52ae46ca00a-kube-api-access-zgb5s\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.731868 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.731555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3adf443a-f972-4340-980e-a52ae46ca00a-webhook-cert\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.733892 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.733869    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3adf443a-f972-4340-980e-a52ae46ca00a-webhook-cert\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.742125 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.742106    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgb5s\" (UniqueName: \"kubernetes.io/projected/3adf443a-f972-4340-980e-a52ae46ca00a-kube-api-access-zgb5s\") pod \"maas-controller-57c89485cc-pfqzg\" (UID: \"3adf443a-f972-4340-980e-a52ae46ca00a\") " pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.834680 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.834627    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:04.964175 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:04.964150    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c89485cc-pfqzg"]
Jul 07 19:14:04.966319 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:14:04.966288    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adf443a_f972_4340_980e_a52ae46ca00a.slice/crio-19c40bc57e2e0546a63edcf24d8464ae6be2e767533b23dc298540b2a076f883 WatchSource:0}: Error finding container 19c40bc57e2e0546a63edcf24d8464ae6be2e767533b23dc298540b2a076f883: Status 404 returned error can't find the container with id 19c40bc57e2e0546a63edcf24d8464ae6be2e767533b23dc298540b2a076f883
Jul 07 19:14:05.408468 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:05.408439    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fbbfd9a-7951-4602-a47c-0bd1eef6bed7" path="/var/lib/kubelet/pods/2fbbfd9a-7951-4602-a47c-0bd1eef6bed7/volumes"
Jul 07 19:14:05.724341 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:05.724254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-pfqzg" event={"ID":"3adf443a-f972-4340-980e-a52ae46ca00a","Type":"ContainerStarted","Data":"e74db6b130b9aa1faa74db74324c1efc967cb3d573ca522a3e4fb56a8855a0e3"}
Jul 07 19:14:05.724341 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:05.724329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c89485cc-pfqzg" event={"ID":"3adf443a-f972-4340-980e-a52ae46ca00a","Type":"ContainerStarted","Data":"19c40bc57e2e0546a63edcf24d8464ae6be2e767533b23dc298540b2a076f883"}
Jul 07 19:14:05.724668 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:05.724379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:14:05.745034 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:05.744994    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57c89485cc-pfqzg" podStartSLOduration=1.318497449 podStartE2EDuration="1.744981534s" totalImagesPullingTime="426.484085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:05.743623957 +0000 UTC m=+811.463423147" watchObservedRunningTime="2026-07-07 19:14:05.744981534 +0000 UTC m=+811.464780766"
Jul 07 19:14:16.736300 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:14:16.736272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57c89485cc-pfqzg"
Jul 07 19:15:00.157590 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.157552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-mfx64"]
Jul 07 19:15:00.159940 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.159915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:00.170552 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.170531    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-mfx64"]
Jul 07 19:15:00.245562 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.245531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvmz\" (UniqueName: \"kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz\") pod \"maas-api-key-cleanup-29724195-mfx64\" (UID: \"1994c35a-0492-4701-a373-97478b35accf\") " pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:00.346976 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.346937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvmz\" (UniqueName: \"kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz\") pod \"maas-api-key-cleanup-29724195-mfx64\" (UID: \"1994c35a-0492-4701-a373-97478b35accf\") " pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:00.369816 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.369786    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvmz\" (UniqueName: \"kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz\") pod \"maas-api-key-cleanup-29724195-mfx64\" (UID: \"1994c35a-0492-4701-a373-97478b35accf\") " pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:00.470283 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.470220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:00.598259 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.598227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724195-mfx64"]
Jul 07 19:15:00.601850 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:15:00.601823    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994c35a_0492_4701_a373_97478b35accf.slice/crio-501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734 WatchSource:0}: Error finding container 501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734: Status 404 returned error can't find the container with id 501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734
Jul 07 19:15:00.969165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:00.969132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64" event={"ID":"1994c35a-0492-4701-a373-97478b35accf","Type":"ContainerStarted","Data":"501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734"}
Jul 07 19:15:03.984746 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:03.984694    2560 generic.go:356] "Generic (PLEG): container finished" podID="1994c35a-0492-4701-a373-97478b35accf" containerID="97472afb2ea6c14b176bcb21d8b98e07ef543dd935416e7791f24c3ec4c279f3" exitCode=0
Jul 07 19:15:03.985079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:03.984766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64" event={"ID":"1994c35a-0492-4701-a373-97478b35accf","Type":"ContainerDied","Data":"97472afb2ea6c14b176bcb21d8b98e07ef543dd935416e7791f24c3ec4c279f3"}
Jul 07 19:15:05.119488 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.119464    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:05.188638 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.188611    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqvmz\" (UniqueName: \"kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz\") pod \"1994c35a-0492-4701-a373-97478b35accf\" (UID: \"1994c35a-0492-4701-a373-97478b35accf\") "
Jul 07 19:15:05.190613 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.190579    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz" (OuterVolumeSpecName: "kube-api-access-vqvmz") pod "1994c35a-0492-4701-a373-97478b35accf" (UID: "1994c35a-0492-4701-a373-97478b35accf"). InnerVolumeSpecName "kube-api-access-vqvmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:15:05.289165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.289141    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvmz\" (UniqueName: \"kubernetes.io/projected/1994c35a-0492-4701-a373-97478b35accf-kube-api-access-vqvmz\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:15:05.995225 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.995196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64"
Jul 07 19:15:05.995392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.995194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724195-mfx64" event={"ID":"1994c35a-0492-4701-a373-97478b35accf","Type":"ContainerDied","Data":"501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734"}
Jul 07 19:15:05.995392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:15:05.995327    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501fcef77cabf5f3267b19e6ceeda5ce141a985f38873e2f1d786d744953c734"
Jul 07 19:18:54.894880 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:54.894794    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:18:54.895394 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:54.895021    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" podUID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" containerName="manager" containerID="cri-o://dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db" gracePeriod=10
Jul 07 19:18:57.447677 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.447652    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:18:57.584663 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.584591    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume\") pod \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") "
Jul 07 19:18:57.584809 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.584684    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svcwm\" (UniqueName: \"kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm\") pod \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\" (UID: \"f8b578a3-af94-4446-b5e2-ac8d8bda60f7\") "
Jul 07 19:18:57.585059 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.585035    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f8b578a3-af94-4446-b5e2-ac8d8bda60f7" (UID: "f8b578a3-af94-4446-b5e2-ac8d8bda60f7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:18:57.586687 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.586668    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm" (OuterVolumeSpecName: "kube-api-access-svcwm") pod "f8b578a3-af94-4446-b5e2-ac8d8bda60f7" (UID: "f8b578a3-af94-4446-b5e2-ac8d8bda60f7"). InnerVolumeSpecName "kube-api-access-svcwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:18:57.686188 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.686165    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svcwm\" (UniqueName: \"kubernetes.io/projected/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-kube-api-access-svcwm\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:18:57.686188 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:57.686185    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8b578a3-af94-4446-b5e2-ac8d8bda60f7-extensions-socket-volume\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:18:58.027289 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.027258    2560 generic.go:356] "Generic (PLEG): container finished" podID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" containerID="dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db" exitCode=0
Jul 07 19:18:58.027443 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.027320    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"
Jul 07 19:18:58.027443 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.027349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" event={"ID":"f8b578a3-af94-4446-b5e2-ac8d8bda60f7","Type":"ContainerDied","Data":"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"}
Jul 07 19:18:58.027443 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.027427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7" event={"ID":"f8b578a3-af94-4446-b5e2-ac8d8bda60f7","Type":"ContainerDied","Data":"110ea57b7d871109e2edab5aaefa3671baa7fbdeb6a56033e0d782f6a3d769fa"}
Jul 07 19:18:58.027541 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.027451    2560 scope.go:117] "RemoveContainer" containerID="dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"
Jul 07 19:18:58.036799 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.036776    2560 scope.go:117] "RemoveContainer" containerID="dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"
Jul 07 19:18:58.037031 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:18:58.037012    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db\": container with ID starting with dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db not found: ID does not exist" containerID="dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"
Jul 07 19:18:58.037088 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.037040    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db"} err="failed to get container status \"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db\": rpc error: code = NotFound desc = could not find container \"dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db\": container with ID starting with dea27bdc0aaeda894dbb278761ef5605e59ad22b4cb821c8e639cd22600351db not found: ID does not exist"
Jul 07 19:18:58.051639 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.051610    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:18:58.057782 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:58.057763    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tk9j7"]
Jul 07 19:18:59.408111 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:18:59.408081    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" path="/var/lib/kubelet/pods/f8b578a3-af94-4446-b5e2-ac8d8bda60f7/volumes"
Jul 07 19:19:56.078893 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.078853    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"]
Jul 07 19:19:56.079274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079181    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1994c35a-0492-4701-a373-97478b35accf" containerName="cleanup"
Jul 07 19:19:56.079274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079193    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1994c35a-0492-4701-a373-97478b35accf" containerName="cleanup"
Jul 07 19:19:56.079274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079215    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" containerName="manager"
Jul 07 19:19:56.079274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079220    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" containerName="manager"
Jul 07 19:19:56.079274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079274    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b578a3-af94-4446-b5e2-ac8d8bda60f7" containerName="manager"
Jul 07 19:19:56.079430 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.079284    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1994c35a-0492-4701-a373-97478b35accf" containerName="cleanup"
Jul 07 19:19:56.081013 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.080997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.083801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.083778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 19:19:56.083921 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.083837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pxtsn\""
Jul 07 19:19:56.084294 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.084275    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 19:19:56.098532 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.098512    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"]
Jul 07 19:19:56.212278 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.212256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bca43827-53bb-4b0e-bef7-83fea49d25e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.212384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.212291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2zt\" (UniqueName: \"kubernetes.io/projected/bca43827-53bb-4b0e-bef7-83fea49d25e3-kube-api-access-lq2zt\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.313120 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.313091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2zt\" (UniqueName: \"kubernetes.io/projected/bca43827-53bb-4b0e-bef7-83fea49d25e3-kube-api-access-lq2zt\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.313209 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.313169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bca43827-53bb-4b0e-bef7-83fea49d25e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.313471 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.313455    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bca43827-53bb-4b0e-bef7-83fea49d25e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.323771 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.323753    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2zt\" (UniqueName: \"kubernetes.io/projected/bca43827-53bb-4b0e-bef7-83fea49d25e3-kube-api-access-lq2zt\") pod \"kuadrant-operator-controller-manager-86878c769f-ct4vk\" (UID: \"bca43827-53bb-4b0e-bef7-83fea49d25e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.391343 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.391273    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:56.523267 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:56.523244    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"]
Jul 07 19:19:56.526081 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:19:56.526050    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca43827_53bb_4b0e_bef7_83fea49d25e3.slice/crio-ebbe6560a1fab9ec58db1c9e503267a56eb7d0c46abef831a95b15b559681ede WatchSource:0}: Error finding container ebbe6560a1fab9ec58db1c9e503267a56eb7d0c46abef831a95b15b559681ede: Status 404 returned error can't find the container with id ebbe6560a1fab9ec58db1c9e503267a56eb7d0c46abef831a95b15b559681ede
Jul 07 19:19:57.294362 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:57.294325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk" event={"ID":"bca43827-53bb-4b0e-bef7-83fea49d25e3","Type":"ContainerStarted","Data":"9556815f48b36795c39be9de60ee10c0354596605d9e1647be26335889562a59"}
Jul 07 19:19:57.294748 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:57.294381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk" event={"ID":"bca43827-53bb-4b0e-bef7-83fea49d25e3","Type":"ContainerStarted","Data":"ebbe6560a1fab9ec58db1c9e503267a56eb7d0c46abef831a95b15b559681ede"}
Jul 07 19:19:57.294748 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:57.294438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:19:57.336331 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:19:57.336290    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk" podStartSLOduration=1.336276804 podStartE2EDuration="1.336276804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:19:57.335811369 +0000 UTC m=+1163.055610558" watchObservedRunningTime="2026-07-07 19:19:57.336276804 +0000 UTC m=+1163.056075994"
Jul 07 19:20:08.301448 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:20:08.301409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ct4vk"
Jul 07 19:26:22.379467 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.379432    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:22.382546 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.382530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.384999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.384975    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-736468f2-serving-cert\""
Jul 07 19:26:22.395645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.395626    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:22.468478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.468444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.468602 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.468526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25lv\" (UniqueName: \"kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.468602 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.468565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.568944 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.568917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.569064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.568980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.569064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.569012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25lv\" (UniqueName: \"kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.569064 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:22.569056    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-736468f2-serving-cert: secret "maas-api-e2e-ait-736468f2-serving-cert" not found
Jul 07 19:26:22.569165 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:22.569117    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls podName:468b71e3-77bb-4ec7-8995-c1549fdfe90d nodeName:}" failed. No retries permitted until 2026-07-07 19:26:23.069100928 +0000 UTC m=+1548.788900099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls") pod "maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" (UID: "468b71e3-77bb-4ec7-8995-c1549fdfe90d") : secret "maas-api-e2e-ait-736468f2-serving-cert" not found
Jul 07 19:26:22.569595 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.569576    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:22.579278 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:22.579258    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25lv\" (UniqueName: \"kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:23.074008 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:23.073975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:23.076370 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:23.076339    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") pod \"maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") " pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:23.292374 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:23.292345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:23.656065 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:23.656040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:23.660827 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:26:23.660794    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468b71e3_77bb_4ec7_8995_c1549fdfe90d.slice/crio-6c2dd69fb40a9a683753fd5b449eea8cf07ae73057b3d621cc78364f13ad45ea WatchSource:0}: Error finding container 6c2dd69fb40a9a683753fd5b449eea8cf07ae73057b3d621cc78364f13ad45ea: Status 404 returned error can't find the container with id 6c2dd69fb40a9a683753fd5b449eea8cf07ae73057b3d621cc78364f13ad45ea
Jul 07 19:26:23.662513 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:23.662495    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:26:24.005082 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:24.005046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" event={"ID":"468b71e3-77bb-4ec7-8995-c1549fdfe90d","Type":"ContainerStarted","Data":"6c2dd69fb40a9a683753fd5b449eea8cf07ae73057b3d621cc78364f13ad45ea"}
Jul 07 19:26:25.012179 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:25.012139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" event={"ID":"468b71e3-77bb-4ec7-8995-c1549fdfe90d","Type":"ContainerStarted","Data":"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"}
Jul 07 19:26:25.012645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:25.012250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:25.034573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:25.034528    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" podStartSLOduration=2.594070569 podStartE2EDuration="3.034516807s" totalImagesPullingTime="440.446238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:25.034224193 +0000 UTC m=+1550.754023384" watchObservedRunningTime="2026-07-07 19:26:25.034516807 +0000 UTC m=+1550.754316035"
Jul 07 19:26:27.558627 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.558588    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:27.559034 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.558810    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" podUID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" containerName="maas-api" containerID="cri-o://baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df" gracePeriod=30
Jul 07 19:26:27.806360 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.806329    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:27.912989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.912931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca\") pod \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") "
Jul 07 19:26:27.912989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.912965    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") pod \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") "
Jul 07 19:26:27.913162 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.913027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g25lv\" (UniqueName: \"kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv\") pod \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\" (UID: \"468b71e3-77bb-4ec7-8995-c1549fdfe90d\") "
Jul 07 19:26:27.913274 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.913237    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "468b71e3-77bb-4ec7-8995-c1549fdfe90d" (UID: "468b71e3-77bb-4ec7-8995-c1549fdfe90d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:26:27.913392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.913361    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/468b71e3-77bb-4ec7-8995-c1549fdfe90d-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:27.915104 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.915081    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "468b71e3-77bb-4ec7-8995-c1549fdfe90d" (UID: "468b71e3-77bb-4ec7-8995-c1549fdfe90d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:26:27.915170 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:27.915085    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv" (OuterVolumeSpecName: "kube-api-access-g25lv") pod "468b71e3-77bb-4ec7-8995-c1549fdfe90d" (UID: "468b71e3-77bb-4ec7-8995-c1549fdfe90d"). InnerVolumeSpecName "kube-api-access-g25lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:26:28.014263 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.014240    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g25lv\" (UniqueName: \"kubernetes.io/projected/468b71e3-77bb-4ec7-8995-c1549fdfe90d-kube-api-access-g25lv\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:28.014263 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.014262    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/468b71e3-77bb-4ec7-8995-c1549fdfe90d-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:28.026621 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.026593    2560 generic.go:356] "Generic (PLEG): container finished" podID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" containerID="baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df" exitCode=0
Jul 07 19:26:28.026727 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.026646    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"
Jul 07 19:26:28.026727 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.026674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" event={"ID":"468b71e3-77bb-4ec7-8995-c1549fdfe90d","Type":"ContainerDied","Data":"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"}
Jul 07 19:26:28.026805 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.026747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj" event={"ID":"468b71e3-77bb-4ec7-8995-c1549fdfe90d","Type":"ContainerDied","Data":"6c2dd69fb40a9a683753fd5b449eea8cf07ae73057b3d621cc78364f13ad45ea"}
Jul 07 19:26:28.026805 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.026769    2560 scope.go:117] "RemoveContainer" containerID="baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"
Jul 07 19:26:28.035514 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.035496    2560 scope.go:117] "RemoveContainer" containerID="baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"
Jul 07 19:26:28.035774 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:28.035757    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df\": container with ID starting with baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df not found: ID does not exist" containerID="baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"
Jul 07 19:26:28.035844 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.035781    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df"} err="failed to get container status \"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df\": rpc error: code = NotFound desc = could not find container \"baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df\": container with ID starting with baa35c0edfcc472fbcbc80e05da31f77e0bda0eab53b54f19c52e4e7ae4830df not found: ID does not exist"
Jul 07 19:26:28.053822 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.053802    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:28.058516 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:28.058498    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-736468f2-5bc587f6fb-64hdj"]
Jul 07 19:26:29.413511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:29.413433    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" path="/var/lib/kubelet/pods/468b71e3-77bb-4ec7-8995-c1549fdfe90d/volumes"
Jul 07 19:26:35.957670 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.957632    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:35.958130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.957983    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" containerName="maas-api"
Jul 07 19:26:35.958130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.957996    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" containerName="maas-api"
Jul 07 19:26:35.958130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.958048    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="468b71e3-77bb-4ec7-8995-c1549fdfe90d" containerName="maas-api"
Jul 07 19:26:35.961831 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.961811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:35.963994 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.963972    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-114401d3-serving-cert\""
Jul 07 19:26:35.972357 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:35.972300    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:36.080104 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.080074    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.080253 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.080129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.080253 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.080160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh48g\" (UniqueName: \"kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.180903 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.180874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh48g\" (UniqueName: \"kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.181044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.180954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.181044 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.181012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.181166 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:36.181105    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-114401d3-serving-cert: secret "maas-api-e2e-ait-114401d3-serving-cert" not found
Jul 07 19:26:36.181221 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:36.181169    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls podName:0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653 nodeName:}" failed. No retries permitted until 2026-07-07 19:26:36.681149174 +0000 UTC m=+1562.400948360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls") pod "maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" (UID: "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653") : secret "maas-api-e2e-ait-114401d3-serving-cert" not found
Jul 07 19:26:36.181665 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.181638    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.196155 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.196128    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh48g\" (UniqueName: \"kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.685394 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.685358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.687645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.687621    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") pod \"maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") " pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:36.872437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:36.872404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:37.217081 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:37.217028    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:37.219187 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:26:37.219159    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7ab90a_4eb6_46d3_a9bf_e42e9ea29653.slice/crio-81664ab771a8d17c5a4ea30a7ae3f921d6de42d57b51958504a4d7b13bbf88e2 WatchSource:0}: Error finding container 81664ab771a8d17c5a4ea30a7ae3f921d6de42d57b51958504a4d7b13bbf88e2: Status 404 returned error can't find the container with id 81664ab771a8d17c5a4ea30a7ae3f921d6de42d57b51958504a4d7b13bbf88e2
Jul 07 19:26:38.073710 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:38.073653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" event={"ID":"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653","Type":"ContainerStarted","Data":"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"}
Jul 07 19:26:38.073899 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:38.073739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" event={"ID":"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653","Type":"ContainerStarted","Data":"81664ab771a8d17c5a4ea30a7ae3f921d6de42d57b51958504a4d7b13bbf88e2"}
Jul 07 19:26:38.073899 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:38.073773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:38.105815 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:38.105764    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" podStartSLOduration=2.756472267 podStartE2EDuration="3.105749403s" totalImagesPullingTime="349.277136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:38.103052311 +0000 UTC m=+1563.822851512" watchObservedRunningTime="2026-07-07 19:26:38.105749403 +0000 UTC m=+1563.825548593"
Jul 07 19:26:41.115499 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.115466    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:41.116275 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.116244    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" podUID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" containerName="maas-api" containerID="cri-o://1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7" gracePeriod=30
Jul 07 19:26:41.361316 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.361287    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:41.428849 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.428788    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") pod \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") "
Jul 07 19:26:41.428849 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.428844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca\") pod \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") "
Jul 07 19:26:41.429029 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.428918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh48g\" (UniqueName: \"kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g\") pod \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\" (UID: \"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653\") "
Jul 07 19:26:41.429229 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.429199    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" (UID: "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:26:41.430868 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.430847    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" (UID: "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:26:41.430942 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.430859    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g" (OuterVolumeSpecName: "kube-api-access-qh48g") pod "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" (UID: "0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653"). InnerVolumeSpecName "kube-api-access-qh48g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:26:41.529382 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.529358    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:41.529476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.529383    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh48g\" (UniqueName: \"kubernetes.io/projected/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-kube-api-access-qh48g\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:41.529476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:41.529399    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:26:42.093626 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.093591    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" containerID="1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7" exitCode=0
Jul 07 19:26:42.093838 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.093636    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"
Jul 07 19:26:42.093838 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.093677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" event={"ID":"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653","Type":"ContainerDied","Data":"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"}
Jul 07 19:26:42.093838 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.093748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7" event={"ID":"0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653","Type":"ContainerDied","Data":"81664ab771a8d17c5a4ea30a7ae3f921d6de42d57b51958504a4d7b13bbf88e2"}
Jul 07 19:26:42.093838 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.093774    2560 scope.go:117] "RemoveContainer" containerID="1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"
Jul 07 19:26:42.102447 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.102435    2560 scope.go:117] "RemoveContainer" containerID="1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"
Jul 07 19:26:42.102761 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:42.102732    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7\": container with ID starting with 1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7 not found: ID does not exist" containerID="1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"
Jul 07 19:26:42.102863 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.102770    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7"} err="failed to get container status \"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7\": rpc error: code = NotFound desc = could not find container \"1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7\": container with ID starting with 1fa3f850692567bf4a830590522b8237ad48f512e73df0000bdeeda040d218d7 not found: ID does not exist"
Jul 07 19:26:42.115526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.115507    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:42.122973 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:42.122954    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-114401d3-857dc9c97d-cbkw7"]
Jul 07 19:26:43.409242 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:43.409210    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" path="/var/lib/kubelet/pods/0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653/volumes"
Jul 07 19:26:55.454344 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.454312    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:26:55.454807 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.454629    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" containerName="maas-api"
Jul 07 19:26:55.454807 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.454641    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" containerName="maas-api"
Jul 07 19:26:55.454807 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.454720    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7ab90a-4eb6-46d3-a9bf-e42e9ea29653" containerName="maas-api"
Jul 07 19:26:55.458808 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.458779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.464108 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.464087    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-9f830d7b-serving-cert\""
Jul 07 19:26:55.492791 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.492765    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:26:55.530575 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.530544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.530575 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.530580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89hn\" (UniqueName: \"kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.530823 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.530618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.631499 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.631472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.631622 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.631502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89hn\" (UniqueName: \"kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.631622 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.631532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.631693 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:55.631650    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-9f830d7b-serving-cert: secret "maas-api-e2e-derive-9f830d7b-serving-cert" not found
Jul 07 19:26:55.631760 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:26:55.631719    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls podName:603f80fe-5eb5-4898-9e14-b7f2a993f084 nodeName:}" failed. No retries permitted until 2026-07-07 19:26:56.131684606 +0000 UTC m=+1581.851483775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls") pod "maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" (UID: "603f80fe-5eb5-4898-9e14-b7f2a993f084") : secret "maas-api-e2e-derive-9f830d7b-serving-cert" not found
Jul 07 19:26:55.632191 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.632172    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:55.645497 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:55.645470    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89hn\" (UniqueName: \"kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:56.135981 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:56.135939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:56.138336 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:56.138313    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") pod \"maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") " pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:56.368291 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:56.368251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:56.509422 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:56.509236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:26:56.512300 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:26:56.512262    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603f80fe_5eb5_4898_9e14_b7f2a993f084.slice/crio-5bfb0aa1a1a8ff488082ccaf62398ec118c8abe16d5d598dc71e65b553d26ed5 WatchSource:0}: Error finding container 5bfb0aa1a1a8ff488082ccaf62398ec118c8abe16d5d598dc71e65b553d26ed5: Status 404 returned error can't find the container with id 5bfb0aa1a1a8ff488082ccaf62398ec118c8abe16d5d598dc71e65b553d26ed5
Jul 07 19:26:57.163472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:57.163392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" event={"ID":"603f80fe-5eb5-4898-9e14-b7f2a993f084","Type":"ContainerStarted","Data":"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"}
Jul 07 19:26:57.163472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:57.163457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" event={"ID":"603f80fe-5eb5-4898-9e14-b7f2a993f084","Type":"ContainerStarted","Data":"5bfb0aa1a1a8ff488082ccaf62398ec118c8abe16d5d598dc71e65b553d26ed5"}
Jul 07 19:26:57.163638 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:57.163489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:26:57.188614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:26:57.188560    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" podStartSLOduration=1.840814585 podStartE2EDuration="2.188539339s" totalImagesPullingTime="347.724754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:57.186005284 +0000 UTC m=+1582.905804475" watchObservedRunningTime="2026-07-07 19:26:57.188539339 +0000 UTC m=+1582.908338530"
Jul 07 19:27:00.117570 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.117538    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:27:00.118099 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.117769    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" podUID="603f80fe-5eb5-4898-9e14-b7f2a993f084" containerName="maas-api" containerID="cri-o://fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92" gracePeriod=30
Jul 07 19:27:00.361868 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.361839    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:27:00.466885 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.466826    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") pod \"603f80fe-5eb5-4898-9e14-b7f2a993f084\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") "
Jul 07 19:27:00.466983 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.466901    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w89hn\" (UniqueName: \"kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn\") pod \"603f80fe-5eb5-4898-9e14-b7f2a993f084\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") "
Jul 07 19:27:00.466983 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.466934    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca\") pod \"603f80fe-5eb5-4898-9e14-b7f2a993f084\" (UID: \"603f80fe-5eb5-4898-9e14-b7f2a993f084\") "
Jul 07 19:27:00.467287 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.467259    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "603f80fe-5eb5-4898-9e14-b7f2a993f084" (UID: "603f80fe-5eb5-4898-9e14-b7f2a993f084"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:27:00.468832 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.468809    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "603f80fe-5eb5-4898-9e14-b7f2a993f084" (UID: "603f80fe-5eb5-4898-9e14-b7f2a993f084"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:27:00.468925 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.468867    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn" (OuterVolumeSpecName: "kube-api-access-w89hn") pod "603f80fe-5eb5-4898-9e14-b7f2a993f084" (UID: "603f80fe-5eb5-4898-9e14-b7f2a993f084"). InnerVolumeSpecName "kube-api-access-w89hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:27:00.568385 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.568357    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89hn\" (UniqueName: \"kubernetes.io/projected/603f80fe-5eb5-4898-9e14-b7f2a993f084-kube-api-access-w89hn\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:27:00.568385 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.568382    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603f80fe-5eb5-4898-9e14-b7f2a993f084-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:27:00.568542 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:00.568391    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603f80fe-5eb5-4898-9e14-b7f2a993f084-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:27:01.182693 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.182658    2560 generic.go:356] "Generic (PLEG): container finished" podID="603f80fe-5eb5-4898-9e14-b7f2a993f084" containerID="fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92" exitCode=0
Jul 07 19:27:01.183060 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.182718    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"
Jul 07 19:27:01.183060 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.182747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" event={"ID":"603f80fe-5eb5-4898-9e14-b7f2a993f084","Type":"ContainerDied","Data":"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"}
Jul 07 19:27:01.183060 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.182801    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td" event={"ID":"603f80fe-5eb5-4898-9e14-b7f2a993f084","Type":"ContainerDied","Data":"5bfb0aa1a1a8ff488082ccaf62398ec118c8abe16d5d598dc71e65b553d26ed5"}
Jul 07 19:27:01.183060 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.182825    2560 scope.go:117] "RemoveContainer" containerID="fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"
Jul 07 19:27:01.191435 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.191417    2560 scope.go:117] "RemoveContainer" containerID="fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"
Jul 07 19:27:01.191738 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:27:01.191694    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92\": container with ID starting with fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92 not found: ID does not exist" containerID="fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"
Jul 07 19:27:01.191853 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.191745    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92"} err="failed to get container status \"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92\": rpc error: code = NotFound desc = could not find container \"fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92\": container with ID starting with fdce9c9ddfcfbca3632abf1d6234fecfa5839583e7bf5a891fe27f0d46052d92 not found: ID does not exist"
Jul 07 19:27:01.207249 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.207222    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:27:01.208927 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.208906    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-9f830d7b-5d4bfb6fc6-gx9td"]
Jul 07 19:27:01.408366 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:27:01.408333    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603f80fe-5eb5-4898-9e14-b7f2a993f084" path="/var/lib/kubelet/pods/603f80fe-5eb5-4898-9e14-b7f2a993f084/volumes"
Jul 07 19:28:52.854397 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.854365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:28:52.854856 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.854727    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603f80fe-5eb5-4898-9e14-b7f2a993f084" containerName="maas-api"
Jul 07 19:28:52.854856 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.854739    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="603f80fe-5eb5-4898-9e14-b7f2a993f084" containerName="maas-api"
Jul 07 19:28:52.854856 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.854801    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="603f80fe-5eb5-4898-9e14-b7f2a993f084" containerName="maas-api"
Jul 07 19:28:52.857573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.857555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:52.863021 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.862993    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-85ca1257-serving-cert\""
Jul 07 19:28:52.872317 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.872289    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:28:52.905603 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.905571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:52.905739 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.905661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6zm\" (UniqueName: \"kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:52.905788 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:52.905732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.006801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.006767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.006944 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.006861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6zm\" (UniqueName: \"kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.006944 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.006897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.007426 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.007403    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.009246 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.009225    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.017403 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.017381    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6zm\" (UniqueName: \"kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm\") pod \"maas-api-e2e-mt-85ca1257-797f787495-nv9d7\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") " pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.167346 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.167289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:53.309926 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.309894    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:28:53.313952 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:28:53.313924    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cb0275_e283_4cb5_b46e_320732efeb0b.slice/crio-4a2e5458f78eebbad0869cb5b8780d547a33a0ad912f4e74f2426818d35366df WatchSource:0}: Error finding container 4a2e5458f78eebbad0869cb5b8780d547a33a0ad912f4e74f2426818d35366df: Status 404 returned error can't find the container with id 4a2e5458f78eebbad0869cb5b8780d547a33a0ad912f4e74f2426818d35366df
Jul 07 19:28:53.694751 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:53.694718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" event={"ID":"50cb0275-e283-4cb5-b46e-320732efeb0b","Type":"ContainerStarted","Data":"4a2e5458f78eebbad0869cb5b8780d547a33a0ad912f4e74f2426818d35366df"}
Jul 07 19:28:54.701726 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:54.701664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" event={"ID":"50cb0275-e283-4cb5-b46e-320732efeb0b","Type":"ContainerStarted","Data":"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"}
Jul 07 19:28:54.702091 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:54.701755    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:28:54.721031 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:28:54.720987    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" podStartSLOduration=2.098901937 podStartE2EDuration="2.720972897s" totalImagesPullingTime="622.07096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:28:54.719389468 +0000 UTC m=+1700.439188658" watchObservedRunningTime="2026-07-07 19:28:54.720972897 +0000 UTC m=+1700.440772087"
Jul 07 19:29:00.711499 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:00.711467    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:29:06.377218 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.377180    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:29:06.377587 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.377430    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" podUID="50cb0275-e283-4cb5-b46e-320732efeb0b" containerName="maas-api" containerID="cri-o://0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806" gracePeriod=30
Jul 07 19:29:06.628077 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.628022    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:29:06.715438 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.715410    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg6zm\" (UniqueName: \"kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm\") pod \"50cb0275-e283-4cb5-b46e-320732efeb0b\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") "
Jul 07 19:29:06.715538 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.715495    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca\") pod \"50cb0275-e283-4cb5-b46e-320732efeb0b\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") "
Jul 07 19:29:06.715538 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.715522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls\") pod \"50cb0275-e283-4cb5-b46e-320732efeb0b\" (UID: \"50cb0275-e283-4cb5-b46e-320732efeb0b\") "
Jul 07 19:29:06.715926 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.715899    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50cb0275-e283-4cb5-b46e-320732efeb0b" (UID: "50cb0275-e283-4cb5-b46e-320732efeb0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:29:06.717486 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.717458    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50cb0275-e283-4cb5-b46e-320732efeb0b" (UID: "50cb0275-e283-4cb5-b46e-320732efeb0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:29:06.717573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.717505    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm" (OuterVolumeSpecName: "kube-api-access-kg6zm") pod "50cb0275-e283-4cb5-b46e-320732efeb0b" (UID: "50cb0275-e283-4cb5-b46e-320732efeb0b"). InnerVolumeSpecName "kube-api-access-kg6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:29:06.756662 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.756635    2560 generic.go:356] "Generic (PLEG): container finished" podID="50cb0275-e283-4cb5-b46e-320732efeb0b" containerID="0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806" exitCode=0
Jul 07 19:29:06.756784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.756674    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"
Jul 07 19:29:06.756784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.756725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" event={"ID":"50cb0275-e283-4cb5-b46e-320732efeb0b","Type":"ContainerDied","Data":"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"}
Jul 07 19:29:06.756892 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.756789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7" event={"ID":"50cb0275-e283-4cb5-b46e-320732efeb0b","Type":"ContainerDied","Data":"4a2e5458f78eebbad0869cb5b8780d547a33a0ad912f4e74f2426818d35366df"}
Jul 07 19:29:06.756892 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.756823    2560 scope.go:117] "RemoveContainer" containerID="0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"
Jul 07 19:29:06.765121 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.765102    2560 scope.go:117] "RemoveContainer" containerID="0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"
Jul 07 19:29:06.765374 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:29:06.765347    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806\": container with ID starting with 0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806 not found: ID does not exist" containerID="0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"
Jul 07 19:29:06.765419 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.765381    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806"} err="failed to get container status \"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806\": rpc error: code = NotFound desc = could not find container \"0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806\": container with ID starting with 0473c089bf652e2c71f3c394f9d29a4a5ecae51cd60fc949dc9b921add80b806 not found: ID does not exist"
Jul 07 19:29:06.780926 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.780907    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:29:06.786127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.786108    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-85ca1257-797f787495-nv9d7"]
Jul 07 19:29:06.816216 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.816196    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6zm\" (UniqueName: \"kubernetes.io/projected/50cb0275-e283-4cb5-b46e-320732efeb0b-kube-api-access-kg6zm\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:29:06.816216 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.816217    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0275-e283-4cb5-b46e-320732efeb0b-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:29:06.816377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:06.816227    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cb0275-e283-4cb5-b46e-320732efeb0b-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:29:07.407762 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:07.407729    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cb0275-e283-4cb5-b46e-320732efeb0b" path="/var/lib/kubelet/pods/50cb0275-e283-4cb5-b46e-320732efeb0b/volumes"
Jul 07 19:29:38.013260 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.013224    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:29:38.013730 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.013552    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cb0275-e283-4cb5-b46e-320732efeb0b" containerName="maas-api"
Jul 07 19:29:38.013730 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.013562    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cb0275-e283-4cb5-b46e-320732efeb0b" containerName="maas-api"
Jul 07 19:29:38.013730 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.013625    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cb0275-e283-4cb5-b46e-320732efeb0b" containerName="maas-api"
Jul 07 19:29:38.016647 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.016621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.018625 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.018602    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-d23e2962-serving-cert\""
Jul 07 19:29:38.030033 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.030010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:29:38.062614 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.062583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.062763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.062636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8knlk\" (UniqueName: \"kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.062763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.062694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.163571 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.163539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.163742 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.163591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8knlk\" (UniqueName: \"kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.163742 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.163626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.163890 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:29:38.163811    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-d23e2962-serving-cert: secret "maas-api-e2e-mt-d23e2962-serving-cert" not found
Jul 07 19:29:38.163947 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:29:38.163921    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls podName:ec5d301a-fa13-460f-97cd-70f94d296dde nodeName:}" failed. No retries permitted until 2026-07-07 19:29:38.663857481 +0000 UTC m=+1744.383656663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls") pod "maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" (UID: "ec5d301a-fa13-460f-97cd-70f94d296dde") : secret "maas-api-e2e-mt-d23e2962-serving-cert" not found
Jul 07 19:29:38.164197 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.164178    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.184859 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.184840    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8knlk\" (UniqueName: \"kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.667372 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.667337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.669694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.669670    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") pod \"maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") " pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:38.927117 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:38.927044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:39.069895 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:39.069869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:29:39.072141 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:29:39.072116    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5d301a_fa13_460f_97cd_70f94d296dde.slice/crio-6209b507eda72f8a31b4496654928831b098b6e8411a0fd5f7497c90ab1d1f63 WatchSource:0}: Error finding container 6209b507eda72f8a31b4496654928831b098b6e8411a0fd5f7497c90ab1d1f63: Status 404 returned error can't find the container with id 6209b507eda72f8a31b4496654928831b098b6e8411a0fd5f7497c90ab1d1f63
Jul 07 19:29:39.904448 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:39.904413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" event={"ID":"ec5d301a-fa13-460f-97cd-70f94d296dde","Type":"ContainerStarted","Data":"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"}
Jul 07 19:29:39.904626 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:39.904470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" event={"ID":"ec5d301a-fa13-460f-97cd-70f94d296dde","Type":"ContainerStarted","Data":"6209b507eda72f8a31b4496654928831b098b6e8411a0fd5f7497c90ab1d1f63"}
Jul 07 19:29:39.904626 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:39.904612    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:29:39.923196 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:39.923152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" podStartSLOduration=2.559678724 podStartE2EDuration="2.923136713s" totalImagesPullingTime="363.457989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:29:39.921919879 +0000 UTC m=+1745.641719072" watchObservedRunningTime="2026-07-07 19:29:39.923136713 +0000 UTC m=+1745.642935902"
Jul 07 19:29:45.915212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:29:45.915184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:30:00.142896 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.142858    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-5m9w4"]
Jul 07 19:30:00.146608 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.146591    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"]
Jul 07 19:30:00.146742 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.146727    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:00.149546 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.149527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:00.155618 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.155544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-5m9w4"]
Jul 07 19:30:00.171746 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.171722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"]
Jul 07 19:30:00.228617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.228586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6rk\" (UniqueName: \"kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk\") pod \"maas-api-key-cleanup-29724210-5m9w4\" (UID: \"1ed15820-3b62-4529-970f-62c23b63bb4f\") " pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:00.228762 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.228622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2hv\" (UniqueName: \"kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv\") pod \"maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24\" (UID: \"3221a05d-c342-4574-9fd1-3d66e07c8485\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:00.329289 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.329247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6rk\" (UniqueName: \"kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk\") pod \"maas-api-key-cleanup-29724210-5m9w4\" (UID: \"1ed15820-3b62-4529-970f-62c23b63bb4f\") " pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:00.329433 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.329292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2hv\" (UniqueName: \"kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv\") pod \"maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24\" (UID: \"3221a05d-c342-4574-9fd1-3d66e07c8485\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:00.339290 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.339270    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2hv\" (UniqueName: \"kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv\") pod \"maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24\" (UID: \"3221a05d-c342-4574-9fd1-3d66e07c8485\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:00.339290 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.339282    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6rk\" (UniqueName: \"kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk\") pod \"maas-api-key-cleanup-29724210-5m9w4\" (UID: \"1ed15820-3b62-4529-970f-62c23b63bb4f\") " pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:00.460196 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.460145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:00.467181 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.467162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:00.817782 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.817755    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-5m9w4"]
Jul 07 19:30:00.820389 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:30:00.820358    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed15820_3b62_4529_970f_62c23b63bb4f.slice/crio-bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c WatchSource:0}: Error finding container bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c: Status 404 returned error can't find the container with id bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c
Jul 07 19:30:00.842938 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:00.842914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"]
Jul 07 19:30:00.851787 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:30:00.851765    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3221a05d_c342_4574_9fd1_3d66e07c8485.slice/crio-969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972 WatchSource:0}: Error finding container 969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972: Status 404 returned error can't find the container with id 969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972
Jul 07 19:30:01.003853 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:01.003817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4" event={"ID":"1ed15820-3b62-4529-970f-62c23b63bb4f","Type":"ContainerStarted","Data":"bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c"}
Jul 07 19:30:01.004983 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:01.004960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24" event={"ID":"3221a05d-c342-4574-9fd1-3d66e07c8485","Type":"ContainerStarted","Data":"969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972"}
Jul 07 19:30:02.013140 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:02.013105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4" event={"ID":"1ed15820-3b62-4529-970f-62c23b63bb4f","Type":"ContainerStarted","Data":"4e6e6d7ed0fec99fd318c56d13b1c55a76773760566e17fa95fb4f5f73e3c3e0"}
Jul 07 19:30:02.014624 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:02.014599    2560 generic.go:356] "Generic (PLEG): container finished" podID="3221a05d-c342-4574-9fd1-3d66e07c8485" containerID="a90b900b6086664f3b0f258771ff8c373fed5a38278c6c3a8e62e4c5aaa54ca1" exitCode=0
Jul 07 19:30:02.014731 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:02.014643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24" event={"ID":"3221a05d-c342-4574-9fd1-3d66e07c8485","Type":"ContainerDied","Data":"a90b900b6086664f3b0f258771ff8c373fed5a38278c6c3a8e62e4c5aaa54ca1"}
Jul 07 19:30:03.020749 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.020690    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ed15820-3b62-4529-970f-62c23b63bb4f" containerID="4e6e6d7ed0fec99fd318c56d13b1c55a76773760566e17fa95fb4f5f73e3c3e0" exitCode=0
Jul 07 19:30:03.020749 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.020727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4" event={"ID":"1ed15820-3b62-4529-970f-62c23b63bb4f","Type":"ContainerDied","Data":"4e6e6d7ed0fec99fd318c56d13b1c55a76773760566e17fa95fb4f5f73e3c3e0"}
Jul 07 19:30:03.156418 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.156393    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:03.183826 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.183803    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:03.256881 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.256855    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn6rk\" (UniqueName: \"kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk\") pod \"1ed15820-3b62-4529-970f-62c23b63bb4f\" (UID: \"1ed15820-3b62-4529-970f-62c23b63bb4f\") "
Jul 07 19:30:03.256988 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.256916    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw2hv\" (UniqueName: \"kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv\") pod \"3221a05d-c342-4574-9fd1-3d66e07c8485\" (UID: \"3221a05d-c342-4574-9fd1-3d66e07c8485\") "
Jul 07 19:30:03.258905 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.258872    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv" (OuterVolumeSpecName: "kube-api-access-bw2hv") pod "3221a05d-c342-4574-9fd1-3d66e07c8485" (UID: "3221a05d-c342-4574-9fd1-3d66e07c8485"). InnerVolumeSpecName "kube-api-access-bw2hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:30:03.259027 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.258925    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk" (OuterVolumeSpecName: "kube-api-access-zn6rk") pod "1ed15820-3b62-4529-970f-62c23b63bb4f" (UID: "1ed15820-3b62-4529-970f-62c23b63bb4f"). InnerVolumeSpecName "kube-api-access-zn6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:30:03.357767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.357692    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6rk\" (UniqueName: \"kubernetes.io/projected/1ed15820-3b62-4529-970f-62c23b63bb4f-kube-api-access-zn6rk\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:30:03.357767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:03.357736    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2hv\" (UniqueName: \"kubernetes.io/projected/3221a05d-c342-4574-9fd1-3d66e07c8485-kube-api-access-bw2hv\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:30:04.026259 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.026230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4"
Jul 07 19:30:04.026647 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.026257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-5m9w4" event={"ID":"1ed15820-3b62-4529-970f-62c23b63bb4f","Type":"ContainerDied","Data":"bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c"}
Jul 07 19:30:04.026647 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.026306    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bba98577f0a8085785fc086e306afbb4bd9efce4f50940c09393eb0da5e85c6c"
Jul 07 19:30:04.027605 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.027584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24" event={"ID":"3221a05d-c342-4574-9fd1-3d66e07c8485","Type":"ContainerDied","Data":"969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972"}
Jul 07 19:30:04.027605 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.027602    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"
Jul 07 19:30:04.027776 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:04.027625    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969b23be0c67490bc1d0fd86337b32e5ae377f77ae8d2240d0ab258b43107972"
Jul 07 19:30:26.406527 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.406437    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:30:26.407086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.406671    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" podUID="ec5d301a-fa13-460f-97cd-70f94d296dde" containerName="maas-api" containerID="cri-o://00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec" gracePeriod=30
Jul 07 19:30:26.447229 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.447204    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"]
Jul 07 19:30:26.451472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.451450    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-d23e2962-29724210-v5b24"]
Jul 07 19:30:26.657669 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.657603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:30:26.735092 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.735067    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca\") pod \"ec5d301a-fa13-460f-97cd-70f94d296dde\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") "
Jul 07 19:30:26.735212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.735107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") pod \"ec5d301a-fa13-460f-97cd-70f94d296dde\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") "
Jul 07 19:30:26.735252 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.735219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8knlk\" (UniqueName: \"kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk\") pod \"ec5d301a-fa13-460f-97cd-70f94d296dde\" (UID: \"ec5d301a-fa13-460f-97cd-70f94d296dde\") "
Jul 07 19:30:26.735402 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.735373    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec5d301a-fa13-460f-97cd-70f94d296dde" (UID: "ec5d301a-fa13-460f-97cd-70f94d296dde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:30:26.735512 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.735500    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d301a-fa13-460f-97cd-70f94d296dde-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:30:26.737029 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.736998    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec5d301a-fa13-460f-97cd-70f94d296dde" (UID: "ec5d301a-fa13-460f-97cd-70f94d296dde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:30:26.737113 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.737089    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk" (OuterVolumeSpecName: "kube-api-access-8knlk") pod "ec5d301a-fa13-460f-97cd-70f94d296dde" (UID: "ec5d301a-fa13-460f-97cd-70f94d296dde"). InnerVolumeSpecName "kube-api-access-8knlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:30:26.836437 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.836415    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8knlk\" (UniqueName: \"kubernetes.io/projected/ec5d301a-fa13-460f-97cd-70f94d296dde-kube-api-access-8knlk\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:30:26.836539 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:26.836440    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5d301a-fa13-460f-97cd-70f94d296dde-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:30:27.133170 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.133135    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec5d301a-fa13-460f-97cd-70f94d296dde" containerID="00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec" exitCode=0
Jul 07 19:30:27.133326 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.133179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" event={"ID":"ec5d301a-fa13-460f-97cd-70f94d296dde","Type":"ContainerDied","Data":"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"}
Jul 07 19:30:27.133326 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.133216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6" event={"ID":"ec5d301a-fa13-460f-97cd-70f94d296dde","Type":"ContainerDied","Data":"6209b507eda72f8a31b4496654928831b098b6e8411a0fd5f7497c90ab1d1f63"}
Jul 07 19:30:27.133326 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.133236    2560 scope.go:117] "RemoveContainer" containerID="00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"
Jul 07 19:30:27.133326 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.133181    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"
Jul 07 19:30:27.143611 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.143497    2560 scope.go:117] "RemoveContainer" containerID="00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"
Jul 07 19:30:27.143905 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:30:27.143882    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec\": container with ID starting with 00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec not found: ID does not exist" containerID="00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"
Jul 07 19:30:27.143985 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.143918    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec"} err="failed to get container status \"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec\": rpc error: code = NotFound desc = could not find container \"00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec\": container with ID starting with 00b0a6b99c265241ce6061e7b3c28b01d662ffb0c9956fc8cbf63d701f359aec not found: ID does not exist"
Jul 07 19:30:27.159309 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.159288    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:30:27.165864 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.165842    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-d23e2962-56cc5c6b5d-tdph6"]
Jul 07 19:30:27.408162 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.408075    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3221a05d-c342-4574-9fd1-3d66e07c8485" path="/var/lib/kubelet/pods/3221a05d-c342-4574-9fd1-3d66e07c8485/volumes"
Jul 07 19:30:27.408620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:30:27.408585    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5d301a-fa13-460f-97cd-70f94d296dde" path="/var/lib/kubelet/pods/ec5d301a-fa13-460f-97cd-70f94d296dde/volumes"
Jul 07 19:32:30.890804 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.890771    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891122    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3221a05d-c342-4574-9fd1-3d66e07c8485" containerName="cleanup"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891134    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3221a05d-c342-4574-9fd1-3d66e07c8485" containerName="cleanup"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5d301a-fa13-460f-97cd-70f94d296dde" containerName="maas-api"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891156    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5d301a-fa13-460f-97cd-70f94d296dde" containerName="maas-api"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891167    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed15820-3b62-4529-970f-62c23b63bb4f" containerName="cleanup"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891172    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed15820-3b62-4529-970f-62c23b63bb4f" containerName="cleanup"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891233    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5d301a-fa13-460f-97cd-70f94d296dde" containerName="maas-api"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891244    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3221a05d-c342-4574-9fd1-3d66e07c8485" containerName="cleanup"
Jul 07 19:32:30.891303 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.891249    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed15820-3b62-4529-970f-62c23b63bb4f" containerName="cleanup"
Jul 07 19:32:30.894345 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.894329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:30.896763 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.896734    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-a17037-serving-cert\""
Jul 07 19:32:30.906339 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.906313    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:32:30.982914 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.982880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:30.983052 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.982955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:30.983052 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:30.983028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9g66\" (UniqueName: \"kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.084004 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.083972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.084133 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.084033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.084133 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.084097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g66\" (UniqueName: \"kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.084133 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:32:31.084121    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-a17037-serving-cert: secret "maas-api-e2e-inf-a-a17037-serving-cert" not found
Jul 07 19:32:31.084296 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:32:31.084187    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls podName:863de4d4-c9ec-4489-8170-a2b04695c49e nodeName:}" failed. No retries permitted until 2026-07-07 19:32:31.584170959 +0000 UTC m=+1917.303970126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls") pod "maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" (UID: "863de4d4-c9ec-4489-8170-a2b04695c49e") : secret "maas-api-e2e-inf-a-a17037-serving-cert" not found
Jul 07 19:32:31.084747 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.084727    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.095138 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.095118    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9g66\" (UniqueName: \"kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.589188 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.589155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.591556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.591528    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") " pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.805049 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.805008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:31.938974 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.938945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:32:31.941095 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:32:31.941072    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863de4d4_c9ec_4489_8170_a2b04695c49e.slice/crio-c89a77855b8f77492cc2f3872dc5ce8fbb2c345476559d8e4a9c90dd1d55f297 WatchSource:0}: Error finding container c89a77855b8f77492cc2f3872dc5ce8fbb2c345476559d8e4a9c90dd1d55f297: Status 404 returned error can't find the container with id c89a77855b8f77492cc2f3872dc5ce8fbb2c345476559d8e4a9c90dd1d55f297
Jul 07 19:32:31.942478 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:31.942462    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:32:32.696304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:32.696225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" event={"ID":"863de4d4-c9ec-4489-8170-a2b04695c49e","Type":"ContainerStarted","Data":"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"}
Jul 07 19:32:32.696304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:32.696286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" event={"ID":"863de4d4-c9ec-4489-8170-a2b04695c49e","Type":"ContainerStarted","Data":"c89a77855b8f77492cc2f3872dc5ce8fbb2c345476559d8e4a9c90dd1d55f297"}
Jul 07 19:32:32.696498 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:32.696352    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:32.718984 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:32.718938    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" podStartSLOduration=2.343102229 podStartE2EDuration="2.718923028s" totalImagesPullingTime="375.820799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:32.717481483 +0000 UTC m=+1918.437280673" watchObservedRunningTime="2026-07-07 19:32:32.718923028 +0000 UTC m=+1918.438722217"
Jul 07 19:32:38.706111 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:38.706085    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:32:42.305401 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.305364    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:32:42.308925 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.308891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.311107 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.310978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-815c5c-serving-cert\""
Jul 07 19:32:42.319927 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.319896    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:32:42.376219 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.376193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrcn\" (UniqueName: \"kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.376361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.376251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.376361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.376336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.476911 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.476887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.477064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.476940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.477064 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:32:42.477016    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-815c5c-serving-cert: secret "maas-api-e2e-inf-b-815c5c-serving-cert" not found
Jul 07 19:32:42.477064 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:32:42.477056    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls podName:2e0e81db-5c20-426e-8ba2-86303369c535 nodeName:}" failed. No retries permitted until 2026-07-07 19:32:42.977043212 +0000 UTC m=+1928.696842380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls") pod "maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" (UID: "2e0e81db-5c20-426e-8ba2-86303369c535") : secret "maas-api-e2e-inf-b-815c5c-serving-cert" not found
Jul 07 19:32:42.477064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.477053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrcn\" (UniqueName: \"kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.477543 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.477524    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.495457 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.495430    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrcn\" (UniqueName: \"kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.980903 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.980866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:42.983271 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:42.983242    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") pod \"maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") " pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:43.222917 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:43.222885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:43.359162 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:43.359084    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:32:43.361484 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:32:43.361454    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0e81db_5c20_426e_8ba2_86303369c535.slice/crio-deda914495ae78ed4ea4a27c594f7afd9c092658fdba1587a043b9d30d9ba0f3 WatchSource:0}: Error finding container deda914495ae78ed4ea4a27c594f7afd9c092658fdba1587a043b9d30d9ba0f3: Status 404 returned error can't find the container with id deda914495ae78ed4ea4a27c594f7afd9c092658fdba1587a043b9d30d9ba0f3
Jul 07 19:32:43.747556 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:43.747519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" event={"ID":"2e0e81db-5c20-426e-8ba2-86303369c535","Type":"ContainerStarted","Data":"deda914495ae78ed4ea4a27c594f7afd9c092658fdba1587a043b9d30d9ba0f3"}
Jul 07 19:32:44.753282 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:44.753243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" event={"ID":"2e0e81db-5c20-426e-8ba2-86303369c535","Type":"ContainerStarted","Data":"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"}
Jul 07 19:32:44.753658 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:44.753322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:32:44.772020 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:44.771975    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" podStartSLOduration=2.347926084 podStartE2EDuration="2.771959249s" totalImagesPullingTime="424.033165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:44.771099734 +0000 UTC m=+1930.490898918" watchObservedRunningTime="2026-07-07 19:32:44.771959249 +0000 UTC m=+1930.491758438"
Jul 07 19:32:50.762821 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:32:50.762790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:33:29.260958 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.260884    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:33:29.264368 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.264348    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.267275 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.267252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-815c5c\"/\"default-dockercfg-25hkz\""
Jul 07 19:33:29.267275 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.267262    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-815c5c\"/\"kube-root-ca.crt\""
Jul 07 19:33:29.267417 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.267299    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-815c5c\"/\"openshift-service-ca.crt\""
Jul 07 19:33:29.267417 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.267251    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-815c5c\"/\"test-model-815c5c-kserve-self-signed-certs\""
Jul 07 19:33:29.280132 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.280110    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:33:29.335440 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.335588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ktx\" (UniqueName: \"kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.335588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.335588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.335588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.335781 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.335593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436082 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ktx\" (UniqueName: \"kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436255 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436510 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436485    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436549 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436499    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.436602 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.436585    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.438436 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.438418    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.438712 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.438680    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.445999 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.445977    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ktx\" (UniqueName: \"kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx\") pod \"test-model-815c5c-kserve-647cbcd999-r6x2n\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") " pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.573207 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.573130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:29.710005 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.709979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:33:29.711854 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:33:29.711824    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb06c60d7_6e88_44eb_8a61_b663c3702468.slice/crio-19456407d82e533a2474fa25fab03460dfba6d8b1b0522211d5a00b8bfe09e89 WatchSource:0}: Error finding container 19456407d82e533a2474fa25fab03460dfba6d8b1b0522211d5a00b8bfe09e89: Status 404 returned error can't find the container with id 19456407d82e533a2474fa25fab03460dfba6d8b1b0522211d5a00b8bfe09e89
Jul 07 19:33:29.956191 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.956109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" event={"ID":"b06c60d7-6e88-44eb-8a61-b663c3702468","Type":"ContainerStarted","Data":"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"}
Jul 07 19:33:29.956191 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.956151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" event={"ID":"b06c60d7-6e88-44eb-8a61-b663c3702468","Type":"ContainerStarted","Data":"19456407d82e533a2474fa25fab03460dfba6d8b1b0522211d5a00b8bfe09e89"}
Jul 07 19:33:29.974539 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:29.974267    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" podStartSLOduration=0.974250589 podStartE2EDuration="974.250589ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:33:29.973957055 +0000 UTC m=+1975.693756269" watchObservedRunningTime="2026-07-07 19:33:29.974250589 +0000 UTC m=+1975.694049782"
Jul 07 19:33:39.574140 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:39.574106    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:39.574631 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:39.574152    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:39.586627 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:39.586602    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:40.014471 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:40.014446    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:33:51.338289 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:51.338255    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:33:52.315504 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.315469    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:33:52.315735 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.315692    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" podUID="863de4d4-c9ec-4489-8170-a2b04695c49e" containerName="maas-api" containerID="cri-o://5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5" gracePeriod=30
Jul 07 19:33:52.572895 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.572841    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:33:52.621314 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.621283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca\") pod \"863de4d4-c9ec-4489-8170-a2b04695c49e\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") "
Jul 07 19:33:52.621492 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.621352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9g66\" (UniqueName: \"kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66\") pod \"863de4d4-c9ec-4489-8170-a2b04695c49e\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") "
Jul 07 19:33:52.621492 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.621389    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") pod \"863de4d4-c9ec-4489-8170-a2b04695c49e\" (UID: \"863de4d4-c9ec-4489-8170-a2b04695c49e\") "
Jul 07 19:33:52.621779 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.621748    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "863de4d4-c9ec-4489-8170-a2b04695c49e" (UID: "863de4d4-c9ec-4489-8170-a2b04695c49e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:33:52.623965 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.623934    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66" (OuterVolumeSpecName: "kube-api-access-h9g66") pod "863de4d4-c9ec-4489-8170-a2b04695c49e" (UID: "863de4d4-c9ec-4489-8170-a2b04695c49e"). InnerVolumeSpecName "kube-api-access-h9g66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:33:52.624075 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.623963    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "863de4d4-c9ec-4489-8170-a2b04695c49e" (UID: "863de4d4-c9ec-4489-8170-a2b04695c49e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:33:52.722756 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.722726    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9g66\" (UniqueName: \"kubernetes.io/projected/863de4d4-c9ec-4489-8170-a2b04695c49e-kube-api-access-h9g66\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:33:52.722756 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.722756    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863de4d4-c9ec-4489-8170-a2b04695c49e-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:33:52.722915 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:52.722767    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863de4d4-c9ec-4489-8170-a2b04695c49e-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:33:53.066620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.066587    2560 generic.go:356] "Generic (PLEG): container finished" podID="863de4d4-c9ec-4489-8170-a2b04695c49e" containerID="5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5" exitCode=0
Jul 07 19:33:53.066790 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.066630    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"
Jul 07 19:33:53.066790 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.066670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" event={"ID":"863de4d4-c9ec-4489-8170-a2b04695c49e","Type":"ContainerDied","Data":"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"}
Jul 07 19:33:53.066790 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.066739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w" event={"ID":"863de4d4-c9ec-4489-8170-a2b04695c49e","Type":"ContainerDied","Data":"c89a77855b8f77492cc2f3872dc5ce8fbb2c345476559d8e4a9c90dd1d55f297"}
Jul 07 19:33:53.066790 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.066763    2560 scope.go:117] "RemoveContainer" containerID="5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"
Jul 07 19:33:53.076335 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.076319    2560 scope.go:117] "RemoveContainer" containerID="5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"
Jul 07 19:33:53.076599 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:33:53.076579    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5\": container with ID starting with 5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5 not found: ID does not exist" containerID="5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"
Jul 07 19:33:53.076653 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.076607    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5"} err="failed to get container status \"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5\": rpc error: code = NotFound desc = could not find container \"5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5\": container with ID starting with 5e93fbab760ead47c8f0f63a9b89aa78ed1635a5d80a843db23549302223d8b5 not found: ID does not exist"
Jul 07 19:33:53.090694 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.090672    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:33:53.097549 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.097531    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a17037-79c9dbf4bd-mmt9w"]
Jul 07 19:33:53.408221 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:33:53.408155    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863de4d4-c9ec-4489-8170-a2b04695c49e" path="/var/lib/kubelet/pods/863de4d4-c9ec-4489-8170-a2b04695c49e/volumes"
Jul 07 19:34:06.358004 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.357944    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" podUID="b06c60d7-6e88-44eb-8a61-b663c3702468" containerName="main" containerID="cri-o://3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84" gracePeriod=45
Jul 07 19:34:06.603421 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.603395    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:34:06.731695 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.731695 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.731913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.731913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.731913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.731913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731812    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69ktx\" (UniqueName: \"kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx\") pod \"b06c60d7-6e88-44eb-8a61-b663c3702468\" (UID: \"b06c60d7-6e88-44eb-8a61-b663c3702468\") "
Jul 07 19:34:06.732129 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731948    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:34:06.732129 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.731955    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home" (OuterVolumeSpecName: "home") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:34:06.732129 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.732042    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache" (OuterVolumeSpecName: "model-cache") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:34:06.732256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.732138    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-tmp-dir\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:06.732256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.732151    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-home\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:06.732256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.732159    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-model-cache\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:06.733918 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.733885    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm" (OuterVolumeSpecName: "dshm") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:34:06.734037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.733948    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:34:06.734037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.733953    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx" (OuterVolumeSpecName: "kube-api-access-69ktx") pod "b06c60d7-6e88-44eb-8a61-b663c3702468" (UID: "b06c60d7-6e88-44eb-8a61-b663c3702468"). InnerVolumeSpecName "kube-api-access-69ktx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:34:06.833064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.833039    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69ktx\" (UniqueName: \"kubernetes.io/projected/b06c60d7-6e88-44eb-8a61-b663c3702468-kube-api-access-69ktx\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:06.833064 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.833061    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06c60d7-6e88-44eb-8a61-b663c3702468-tls-certs\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:06.833193 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:06.833072    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06c60d7-6e88-44eb-8a61-b663c3702468-dshm\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:07.132366 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.132336    2560 generic.go:356] "Generic (PLEG): container finished" podID="b06c60d7-6e88-44eb-8a61-b663c3702468" containerID="3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84" exitCode=0
Jul 07 19:34:07.132511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.132380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"
Jul 07 19:34:07.132511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.132410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" event={"ID":"b06c60d7-6e88-44eb-8a61-b663c3702468","Type":"ContainerDied","Data":"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"}
Jul 07 19:34:07.132511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.132446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n" event={"ID":"b06c60d7-6e88-44eb-8a61-b663c3702468","Type":"ContainerDied","Data":"19456407d82e533a2474fa25fab03460dfba6d8b1b0522211d5a00b8bfe09e89"}
Jul 07 19:34:07.132511 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.132465    2560 scope.go:117] "RemoveContainer" containerID="3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"
Jul 07 19:34:07.141561 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.141545    2560 scope.go:117] "RemoveContainer" containerID="3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"
Jul 07 19:34:07.141849 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:34:07.141831    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84\": container with ID starting with 3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84 not found: ID does not exist" containerID="3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"
Jul 07 19:34:07.141913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.141858    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84"} err="failed to get container status \"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84\": rpc error: code = NotFound desc = could not find container \"3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84\": container with ID starting with 3d255a62959fd60ea8a3868a0ea2a65d72277b144a076d058300f9c50c3d4d84 not found: ID does not exist"
Jul 07 19:34:07.155881 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.155856    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:34:07.161033 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.161015    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-815c5c/test-model-815c5c-kserve-647cbcd999-r6x2n"]
Jul 07 19:34:07.409057 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:07.408987    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b06c60d7-6e88-44eb-8a61-b663c3702468" path="/var/lib/kubelet/pods/b06c60d7-6e88-44eb-8a61-b663c3702468/volumes"
Jul 07 19:34:35.941356 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:35.941320    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:34:35.941800 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:35.941619    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" podUID="2e0e81db-5c20-426e-8ba2-86303369c535" containerName="maas-api" containerID="cri-o://bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced" gracePeriod=30
Jul 07 19:34:36.186474 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.186446    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:34:36.265595 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.265561    2560 generic.go:356] "Generic (PLEG): container finished" podID="2e0e81db-5c20-426e-8ba2-86303369c535" containerID="bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced" exitCode=0
Jul 07 19:34:36.265774 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.265593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" event={"ID":"2e0e81db-5c20-426e-8ba2-86303369c535","Type":"ContainerDied","Data":"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"}
Jul 07 19:34:36.265774 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.265604    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"
Jul 07 19:34:36.265774 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.265645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h" event={"ID":"2e0e81db-5c20-426e-8ba2-86303369c535","Type":"ContainerDied","Data":"deda914495ae78ed4ea4a27c594f7afd9c092658fdba1587a043b9d30d9ba0f3"}
Jul 07 19:34:36.265774 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.265673    2560 scope.go:117] "RemoveContainer" containerID="bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"
Jul 07 19:34:36.273729 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.273711    2560 scope.go:117] "RemoveContainer" containerID="bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"
Jul 07 19:34:36.273986 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:34:36.273968    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced\": container with ID starting with bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced not found: ID does not exist" containerID="bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"
Jul 07 19:34:36.274035 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.274003    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced"} err="failed to get container status \"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced\": rpc error: code = NotFound desc = could not find container \"bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced\": container with ID starting with bccfa4721f35e35d7f2c967f8f87bfe9b078ee8f4ba6d41eb6feb8566be6eced not found: ID does not exist"
Jul 07 19:34:36.358322 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.358297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca\") pod \"2e0e81db-5c20-426e-8ba2-86303369c535\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") "
Jul 07 19:34:36.358464 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.358395    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hrcn\" (UniqueName: \"kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn\") pod \"2e0e81db-5c20-426e-8ba2-86303369c535\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") "
Jul 07 19:34:36.358464 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.358446    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") pod \"2e0e81db-5c20-426e-8ba2-86303369c535\" (UID: \"2e0e81db-5c20-426e-8ba2-86303369c535\") "
Jul 07 19:34:36.358696 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.358675    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e0e81db-5c20-426e-8ba2-86303369c535" (UID: "2e0e81db-5c20-426e-8ba2-86303369c535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:34:36.360441 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.360416    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e0e81db-5c20-426e-8ba2-86303369c535" (UID: "2e0e81db-5c20-426e-8ba2-86303369c535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:34:36.360520 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.360484    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn" (OuterVolumeSpecName: "kube-api-access-6hrcn") pod "2e0e81db-5c20-426e-8ba2-86303369c535" (UID: "2e0e81db-5c20-426e-8ba2-86303369c535"). InnerVolumeSpecName "kube-api-access-6hrcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:34:36.459743 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.459722    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrcn\" (UniqueName: \"kubernetes.io/projected/2e0e81db-5c20-426e-8ba2-86303369c535-kube-api-access-6hrcn\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:36.459743 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.459743    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0e81db-5c20-426e-8ba2-86303369c535-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:36.459880 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.459752    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0e81db-5c20-426e-8ba2-86303369c535-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:34:36.596115 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.596092    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:34:36.616463 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:36.616437    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-815c5c-5c5ffdb5-f8s9h"]
Jul 07 19:34:37.408434 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:37.408405    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0e81db-5c20-426e-8ba2-86303369c535" path="/var/lib/kubelet/pods/2e0e81db-5c20-426e-8ba2-86303369c535/volumes"
Jul 07 19:34:46.663090 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663057    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663549    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863de4d4-c9ec-4489-8170-a2b04695c49e" containerName="maas-api"
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663569    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="863de4d4-c9ec-4489-8170-a2b04695c49e" containerName="maas-api"
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663587    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0e81db-5c20-426e-8ba2-86303369c535" containerName="maas-api"
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663596    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0e81db-5c20-426e-8ba2-86303369c535" containerName="maas-api"
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663621    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b06c60d7-6e88-44eb-8a61-b663c3702468" containerName="main"
Jul 07 19:34:46.663637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663628    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b06c60d7-6e88-44eb-8a61-b663c3702468" containerName="main"
Jul 07 19:34:46.663984 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663721    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b06c60d7-6e88-44eb-8a61-b663c3702468" containerName="main"
Jul 07 19:34:46.663984 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663736    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="863de4d4-c9ec-4489-8170-a2b04695c49e" containerName="maas-api"
Jul 07 19:34:46.663984 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.663747    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0e81db-5c20-426e-8ba2-86303369c535" containerName="maas-api"
Jul 07 19:34:46.666875 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.666854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.669003 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.668985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-2e6bb8-serving-cert\""
Jul 07 19:34:46.677227 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.677206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:34:46.741740 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.741711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.741862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.741810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.741862 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.741856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvchw\" (UniqueName: \"kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.843030 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.842995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.843172 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.843061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.843172 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.843111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvchw\" (UniqueName: \"kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.843628 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.843604    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.845447 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.845417    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.852526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.852503    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvchw\" (UniqueName: \"kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw\") pod \"maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") " pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:46.977728 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:46.977665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:47.131465 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:47.131442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:34:47.133790 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:34:47.133755    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f487ad5_8745_470e_a468_648305541109.slice/crio-672e5e28d5471b6bc42d2b365d73aa8094d86af9962224568f59ccb3d7273921 WatchSource:0}: Error finding container 672e5e28d5471b6bc42d2b365d73aa8094d86af9962224568f59ccb3d7273921: Status 404 returned error can't find the container with id 672e5e28d5471b6bc42d2b365d73aa8094d86af9962224568f59ccb3d7273921
Jul 07 19:34:47.319371 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:47.319336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" event={"ID":"9f487ad5-8745-470e-a468-648305541109","Type":"ContainerStarted","Data":"672e5e28d5471b6bc42d2b365d73aa8094d86af9962224568f59ccb3d7273921"}
Jul 07 19:34:48.325457 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:48.325420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" event={"ID":"9f487ad5-8745-470e-a468-648305541109","Type":"ContainerStarted","Data":"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"}
Jul 07 19:34:48.325896 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:48.325540    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:48.346095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:48.346048    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" podStartSLOduration=1.9505694340000002 podStartE2EDuration="2.346035528s" totalImagesPullingTime="395.466094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:48.345728944 +0000 UTC m=+2054.065528133" watchObservedRunningTime="2026-07-07 19:34:48.346035528 +0000 UTC m=+2054.065834709"
Jul 07 19:34:54.335435 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.335406    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:34:54.595880 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.595788    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:34:54.600001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.599978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.602560 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.602540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-89748e-serving-cert\""
Jul 07 19:34:54.609452 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.609428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:34:54.704462 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.704436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.704612 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.704499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trtf\" (UniqueName: \"kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.704612 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.704582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.805221 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.805187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trtf\" (UniqueName: \"kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.805361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.805241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.805361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.805275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.806127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.806086    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.807844 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.807821    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.816137 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.816113    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trtf\" (UniqueName: \"kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf\") pod \"maas-api-e2e-api-b-89748e-675c6fbcc4-t896w\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") " pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:54.912716 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:54.912644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:55.051977 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:55.051953    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:34:55.054368 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:34:55.054330    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cdfaf0_706a_4aa4_98bc_c5958644b268.slice/crio-9f7f6266adfcbf411b6cb3ba99db2793117a3e742540d5262f271b26f396f806 WatchSource:0}: Error finding container 9f7f6266adfcbf411b6cb3ba99db2793117a3e742540d5262f271b26f396f806: Status 404 returned error can't find the container with id 9f7f6266adfcbf411b6cb3ba99db2793117a3e742540d5262f271b26f396f806
Jul 07 19:34:55.357282 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:55.357251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" event={"ID":"31cdfaf0-706a-4aa4-98bc-c5958644b268","Type":"ContainerStarted","Data":"9f7f6266adfcbf411b6cb3ba99db2793117a3e742540d5262f271b26f396f806"}
Jul 07 19:34:56.363438 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:56.363402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" event={"ID":"31cdfaf0-706a-4aa4-98bc-c5958644b268","Type":"ContainerStarted","Data":"fc027f897a486a07ad5abf977ed5f6752fa7d7dd05aa4a85e2aac03580690ed0"}
Jul 07 19:34:56.363872 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:56.363537    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:34:56.383367 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:34:56.383253    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" podStartSLOduration=1.97124017 podStartE2EDuration="2.383238335s" totalImagesPullingTime="411.998165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:56.382899534 +0000 UTC m=+2062.102698724" watchObservedRunningTime="2026-07-07 19:34:56.383238335 +0000 UTC m=+2062.103037525"
Jul 07 19:35:02.373716 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:02.373677    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:35:08.003637 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.003601    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:35:08.005653 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.004758    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" podUID="9f487ad5-8745-470e-a468-648305541109" containerName="maas-api" containerID="cri-o://194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece" gracePeriod=30
Jul 07 19:35:08.254636 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.254577    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:35:08.320167 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.320132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca\") pod \"9f487ad5-8745-470e-a468-648305541109\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") "
Jul 07 19:35:08.320314 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.320190    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls\") pod \"9f487ad5-8745-470e-a468-648305541109\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") "
Jul 07 19:35:08.320381 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.320312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvchw\" (UniqueName: \"kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw\") pod \"9f487ad5-8745-470e-a468-648305541109\" (UID: \"9f487ad5-8745-470e-a468-648305541109\") "
Jul 07 19:35:08.320456 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.320436    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f487ad5-8745-470e-a468-648305541109" (UID: "9f487ad5-8745-470e-a468-648305541109"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:35:08.320620 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.320605    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f487ad5-8745-470e-a468-648305541109-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:08.322112 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.322092    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f487ad5-8745-470e-a468-648305541109" (UID: "9f487ad5-8745-470e-a468-648305541109"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:35:08.322188 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.322168    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw" (OuterVolumeSpecName: "kube-api-access-tvchw") pod "9f487ad5-8745-470e-a468-648305541109" (UID: "9f487ad5-8745-470e-a468-648305541109"). InnerVolumeSpecName "kube-api-access-tvchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:35:08.417273 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.417250    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f487ad5-8745-470e-a468-648305541109" containerID="194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece" exitCode=0
Jul 07 19:35:08.417380 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.417297    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"
Jul 07 19:35:08.417380 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.417341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" event={"ID":"9f487ad5-8745-470e-a468-648305541109","Type":"ContainerDied","Data":"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"}
Jul 07 19:35:08.417458 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.417383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc" event={"ID":"9f487ad5-8745-470e-a468-648305541109","Type":"ContainerDied","Data":"672e5e28d5471b6bc42d2b365d73aa8094d86af9962224568f59ccb3d7273921"}
Jul 07 19:35:08.417458 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.417406    2560 scope.go:117] "RemoveContainer" containerID="194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"
Jul 07 19:35:08.421884 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.421865    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f487ad5-8745-470e-a468-648305541109-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:08.421884 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.421885    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvchw\" (UniqueName: \"kubernetes.io/projected/9f487ad5-8745-470e-a468-648305541109-kube-api-access-tvchw\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:08.426333 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.426316    2560 scope.go:117] "RemoveContainer" containerID="194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"
Jul 07 19:35:08.426549 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:35:08.426532    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece\": container with ID starting with 194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece not found: ID does not exist" containerID="194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"
Jul 07 19:35:08.426596 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.426555    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece"} err="failed to get container status \"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece\": rpc error: code = NotFound desc = could not find container \"194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece\": container with ID starting with 194987354bf81e47faf34567fc7feb27c62438c21f41aadd64999535c0908ece not found: ID does not exist"
Jul 07 19:35:08.441968 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.441943    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:35:08.448060 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:08.448040    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2e6bb8-7bb6875c95-8cbtc"]
Jul 07 19:35:09.409367 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:09.409335    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f487ad5-8745-470e-a468-648305541109" path="/var/lib/kubelet/pods/9f487ad5-8745-470e-a468-648305541109/volumes"
Jul 07 19:35:16.310586 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.310552    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:35:16.311177 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.310788    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" podUID="31cdfaf0-706a-4aa4-98bc-c5958644b268" containerName="maas-api" containerID="cri-o://fc027f897a486a07ad5abf977ed5f6752fa7d7dd05aa4a85e2aac03580690ed0" gracePeriod=30
Jul 07 19:35:16.454485 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.454447    2560 generic.go:356] "Generic (PLEG): container finished" podID="31cdfaf0-706a-4aa4-98bc-c5958644b268" containerID="fc027f897a486a07ad5abf977ed5f6752fa7d7dd05aa4a85e2aac03580690ed0" exitCode=0
Jul 07 19:35:16.454616 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.454519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" event={"ID":"31cdfaf0-706a-4aa4-98bc-c5958644b268","Type":"ContainerDied","Data":"fc027f897a486a07ad5abf977ed5f6752fa7d7dd05aa4a85e2aac03580690ed0"}
Jul 07 19:35:16.550692 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.550665    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:35:16.690981 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.690896    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca\") pod \"31cdfaf0-706a-4aa4-98bc-c5958644b268\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") "
Jul 07 19:35:16.691144 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.690993    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2trtf\" (UniqueName: \"kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf\") pod \"31cdfaf0-706a-4aa4-98bc-c5958644b268\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") "
Jul 07 19:35:16.691144 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.691054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls\") pod \"31cdfaf0-706a-4aa4-98bc-c5958644b268\" (UID: \"31cdfaf0-706a-4aa4-98bc-c5958644b268\") "
Jul 07 19:35:16.691346 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.691323    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31cdfaf0-706a-4aa4-98bc-c5958644b268" (UID: "31cdfaf0-706a-4aa4-98bc-c5958644b268"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:35:16.693621 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.693593    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31cdfaf0-706a-4aa4-98bc-c5958644b268" (UID: "31cdfaf0-706a-4aa4-98bc-c5958644b268"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:35:16.693767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.693599    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf" (OuterVolumeSpecName: "kube-api-access-2trtf") pod "31cdfaf0-706a-4aa4-98bc-c5958644b268" (UID: "31cdfaf0-706a-4aa4-98bc-c5958644b268"). InnerVolumeSpecName "kube-api-access-2trtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:35:16.791891 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.791852    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31cdfaf0-706a-4aa4-98bc-c5958644b268-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:16.791891 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.791884    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31cdfaf0-706a-4aa4-98bc-c5958644b268-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:16.791891 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:16.791893    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trtf\" (UniqueName: \"kubernetes.io/projected/31cdfaf0-706a-4aa4-98bc-c5958644b268-kube-api-access-2trtf\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:35:17.459991 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:17.459963    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"
Jul 07 19:35:17.460485 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:17.460004    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w" event={"ID":"31cdfaf0-706a-4aa4-98bc-c5958644b268","Type":"ContainerDied","Data":"9f7f6266adfcbf411b6cb3ba99db2793117a3e742540d5262f271b26f396f806"}
Jul 07 19:35:17.460485 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:17.460063    2560 scope.go:117] "RemoveContainer" containerID="fc027f897a486a07ad5abf977ed5f6752fa7d7dd05aa4a85e2aac03580690ed0"
Jul 07 19:35:17.479916 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:17.479894    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:35:17.485100 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:17.485081    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-89748e-675c6fbcc4-t896w"]
Jul 07 19:35:19.408038 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:19.408004    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cdfaf0-706a-4aa4-98bc-c5958644b268" path="/var/lib/kubelet/pods/31cdfaf0-706a-4aa4-98bc-c5958644b268/volumes"
Jul 07 19:35:26.264865 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.264834    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:35:26.265352 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265333    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cdfaf0-706a-4aa4-98bc-c5958644b268" containerName="maas-api"
Jul 07 19:35:26.265442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265355    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cdfaf0-706a-4aa4-98bc-c5958644b268" containerName="maas-api"
Jul 07 19:35:26.265442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265370    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f487ad5-8745-470e-a468-648305541109" containerName="maas-api"
Jul 07 19:35:26.265442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265378    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f487ad5-8745-470e-a468-648305541109" containerName="maas-api"
Jul 07 19:35:26.265588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265464    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cdfaf0-706a-4aa4-98bc-c5958644b268" containerName="maas-api"
Jul 07 19:35:26.265588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.265477    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f487ad5-8745-470e-a468-648305541109" containerName="maas-api"
Jul 07 19:35:26.269832 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.269811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.271858 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.271836    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-0e4903-serving-cert\""
Jul 07 19:35:26.278213 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.278195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:35:26.363405 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.363369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.363541 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.363407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.363541 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.363493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4frh\" (UniqueName: \"kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.464880 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.464846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.465010 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.464886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.465010 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.464928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4frh\" (UniqueName: \"kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.465010 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:35:26.465001    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-0e4903-serving-cert: secret "maas-api-e2e-shared-a-0e4903-serving-cert" not found
Jul 07 19:35:26.465191 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:35:26.465079    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls podName:be73bbd6-d6dc-4841-9029-cc4816fbfb66 nodeName:}" failed. No retries permitted until 2026-07-07 19:35:26.965060063 +0000 UTC m=+2092.684859236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls") pod "maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" (UID: "be73bbd6-d6dc-4841-9029-cc4816fbfb66") : secret "maas-api-e2e-shared-a-0e4903-serving-cert" not found
Jul 07 19:35:26.465654 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.465634    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.474734 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.474718    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4frh\" (UniqueName: \"kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.969941 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.969907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:26.972463 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:26.972436    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") pod \"maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") " pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:27.180924 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:27.180889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:27.330768 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:27.330742    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:35:27.332851 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:35:27.332825    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe73bbd6_d6dc_4841_9029_cc4816fbfb66.slice/crio-d85b1aa59e69472fe854a7175b4b6af823bd78ac2d850e426ea68329b16a0c57 WatchSource:0}: Error finding container d85b1aa59e69472fe854a7175b4b6af823bd78ac2d850e426ea68329b16a0c57: Status 404 returned error can't find the container with id d85b1aa59e69472fe854a7175b4b6af823bd78ac2d850e426ea68329b16a0c57
Jul 07 19:35:27.506189 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:27.506153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" event={"ID":"be73bbd6-d6dc-4841-9029-cc4816fbfb66","Type":"ContainerStarted","Data":"d85b1aa59e69472fe854a7175b4b6af823bd78ac2d850e426ea68329b16a0c57"}
Jul 07 19:35:28.512994 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:28.512955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" event={"ID":"be73bbd6-d6dc-4841-9029-cc4816fbfb66","Type":"ContainerStarted","Data":"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"}
Jul 07 19:35:28.513419 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:28.513081    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:28.538279 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:28.538234    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" podStartSLOduration=2.153290768 podStartE2EDuration="2.538208313s" totalImagesPullingTime="384.917545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:28.537814342 +0000 UTC m=+2094.257613523" watchObservedRunningTime="2026-07-07 19:35:28.538208313 +0000 UTC m=+2094.258007505"
Jul 07 19:35:33.949976 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:33.949940    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:35:33.953473 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:33.953454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:33.955553 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:33.955532    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-836c8d-serving-cert\""
Jul 07 19:35:33.965588 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:33.965568    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:35:34.132256 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.132220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.132455 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.132388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.132455 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.132426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppm4k\" (UniqueName: \"kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.233165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.233074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.233165 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.233116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppm4k\" (UniqueName: \"kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.233386 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.233166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.233882 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.233851    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.235472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.235449    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.248163 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.248144    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppm4k\" (UniqueName: \"kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k\") pod \"maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") " pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.265017 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.264993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:34.523095 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.523069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:35:34.609476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:34.609449    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:35:34.612013 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:35:34.611986    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4d2c01_eee2_47de_bae1_11d50a59700d.slice/crio-ddd15401e6d57479fd08d9040b0d9258f92c9f02843e3ccacd2df3c014cc8df7 WatchSource:0}: Error finding container ddd15401e6d57479fd08d9040b0d9258f92c9f02843e3ccacd2df3c014cc8df7: Status 404 returned error can't find the container with id ddd15401e6d57479fd08d9040b0d9258f92c9f02843e3ccacd2df3c014cc8df7
Jul 07 19:35:35.546529 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:35.546493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" event={"ID":"6f4d2c01-eee2-47de-bae1-11d50a59700d","Type":"ContainerStarted","Data":"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"}
Jul 07 19:35:35.546898 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:35.546542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" event={"ID":"6f4d2c01-eee2-47de-bae1-11d50a59700d","Type":"ContainerStarted","Data":"ddd15401e6d57479fd08d9040b0d9258f92c9f02843e3ccacd2df3c014cc8df7"}
Jul 07 19:35:35.546898 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:35.546617    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:35:35.567597 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:35.567548    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" podStartSLOduration=2.215725503 podStartE2EDuration="2.567532851s" totalImagesPullingTime="351.807348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:35.564987348 +0000 UTC m=+2101.284786537" watchObservedRunningTime="2026-07-07 19:35:35.567532851 +0000 UTC m=+2101.287332041"
Jul 07 19:35:41.556510 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:35:41.556481    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:36:35.480610 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:36:35.480534    2560 scope.go:117] "RemoveContainer" containerID="a90b900b6086664f3b0f258771ff8c373fed5a38278c6c3a8e62e4c5aaa54ca1"
Jul 07 19:38:38.166389 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.166355    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:38:38.169559 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.169542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.171617 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.171593    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"openshift-service-ca.crt\""
Jul 07 19:38:38.171770 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.171736    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"kube-root-ca.crt\""
Jul 07 19:38:38.171924 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.171906    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"default-dockercfg-b9fbb\""
Jul 07 19:38:38.172847 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.172829    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"sub-test-model-0e4903-kserve-self-signed-certs\""
Jul 07 19:38:38.185102 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.185078    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:38:38.277229 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.277383 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.277456 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.277532 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.277600 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcpxp\" (UniqueName: \"kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.277659 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.277643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378632 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378798 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378798 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378798 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378798 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.378798 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.378752    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcpxp\" (UniqueName: \"kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.379086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.379004    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.379086 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.379071    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.379208 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.379095    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.381008 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.380986    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.381361 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.381339    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.390522 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.390492    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcpxp\" (UniqueName: \"kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp\") pod \"sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") " pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.479043 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.478989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:38.611203 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:38.611169    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:38:38.613970 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:38:38.613936    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd918ca84_d7b6_4ed3_baa7_b7abc4faee24.slice/crio-d9be2a514f81ae120ffddb61aadfe2e61918d18d83f0d88fd9819efe32b26d14 WatchSource:0}: Error finding container d9be2a514f81ae120ffddb61aadfe2e61918d18d83f0d88fd9819efe32b26d14: Status 404 returned error can't find the container with id d9be2a514f81ae120ffddb61aadfe2e61918d18d83f0d88fd9819efe32b26d14
Jul 07 19:38:39.381519 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:39.381478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" event={"ID":"d918ca84-d7b6-4ed3-baa7-b7abc4faee24","Type":"ContainerStarted","Data":"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"}
Jul 07 19:38:39.381883 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:39.381529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" event={"ID":"d918ca84-d7b6-4ed3-baa7-b7abc4faee24","Type":"ContainerStarted","Data":"d9be2a514f81ae120ffddb61aadfe2e61918d18d83f0d88fd9819efe32b26d14"}
Jul 07 19:38:39.402270 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:39.402220    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" podStartSLOduration=1.402202234 podStartE2EDuration="1.402202234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:38:39.401316182 +0000 UTC m=+2285.121115363" watchObservedRunningTime="2026-07-07 19:38:39.402202234 +0000 UTC m=+2285.122001425"
Jul 07 19:38:48.479468 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:48.479420    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:48.479891 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:48.479488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:48.494519 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:48.494491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:38:49.437961 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:38:49.437933    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:39:34.075049 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:34.074975    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:39:49.095393 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.095338    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" podUID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" containerName="main" containerID="cri-o://442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4" gracePeriod=45
Jul 07 19:39:49.347323 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.347272    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:39:49.442499 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442564    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcpxp\" (UniqueName: \"kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442590    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442651    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs\") pod \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\" (UID: \"d918ca84-d7b6-4ed3-baa7-b7abc4faee24\") "
Jul 07 19:39:49.442959 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442773    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:39:49.442959 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442939    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache" (OuterVolumeSpecName: "model-cache") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:39:49.443054 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442951    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home" (OuterVolumeSpecName: "home") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:39:49.443054 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.442961    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tmp-dir\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.444884 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.444858    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:39:49.444970 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.444870    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp" (OuterVolumeSpecName: "kube-api-access-dcpxp") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "kube-api-access-dcpxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:39:49.444970 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.444936    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm" (OuterVolumeSpecName: "dshm") pod "d918ca84-d7b6-4ed3-baa7-b7abc4faee24" (UID: "d918ca84-d7b6-4ed3-baa7-b7abc4faee24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:39:49.544174 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.544153    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-model-cache\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.544174 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.544175    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcpxp\" (UniqueName: \"kubernetes.io/projected/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-kube-api-access-dcpxp\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.544327 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.544185    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-dshm\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.544327 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.544193    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-home\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.544327 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.544202    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d918ca84-d7b6-4ed3-baa7-b7abc4faee24-tls-certs\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:39:49.704714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.704622    2560 generic.go:356] "Generic (PLEG): container finished" podID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" containerID="442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4" exitCode=0
Jul 07 19:39:49.704714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.704666    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"
Jul 07 19:39:49.704714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.704679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" event={"ID":"d918ca84-d7b6-4ed3-baa7-b7abc4faee24","Type":"ContainerDied","Data":"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"}
Jul 07 19:39:49.704938 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.704740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7" event={"ID":"d918ca84-d7b6-4ed3-baa7-b7abc4faee24","Type":"ContainerDied","Data":"d9be2a514f81ae120ffddb61aadfe2e61918d18d83f0d88fd9819efe32b26d14"}
Jul 07 19:39:49.704938 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.704763    2560 scope.go:117] "RemoveContainer" containerID="442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"
Jul 07 19:39:49.713403 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.713386    2560 scope.go:117] "RemoveContainer" containerID="442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"
Jul 07 19:39:49.713661 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:39:49.713641    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4\": container with ID starting with 442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4 not found: ID does not exist" containerID="442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"
Jul 07 19:39:49.713730 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.713670    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4"} err="failed to get container status \"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4\": rpc error: code = NotFound desc = could not find container \"442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4\": container with ID starting with 442391344e77dc9c0b63a94f0c3d323a7a84803c06e920d0a3b185508155b1a4 not found: ID does not exist"
Jul 07 19:39:49.726177 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.726149    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:39:49.731330 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:49.731310    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/sub-test-model-0e4903-kserve-6cf4d9bd77-m24p7"]
Jul 07 19:39:51.408307 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:39:51.408272    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" path="/var/lib/kubelet/pods/d918ca84-d7b6-4ed3-baa7-b7abc4faee24/volumes"
Jul 07 19:40:20.362472 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.362436    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:40:20.362893 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.362836    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" containerName="main"
Jul 07 19:40:20.362893 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.362847    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" containerName="main"
Jul 07 19:40:20.362966 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.362905    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d918ca84-d7b6-4ed3-baa7-b7abc4faee24" containerName="main"
Jul 07 19:40:20.365459 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.365444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.367618 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.367594    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"kube-root-ca.crt\""
Jul 07 19:40:20.367768 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.367666    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"openshift-service-ca.crt\""
Jul 07 19:40:20.367768 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.367688    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"default-dockercfg-frq88\""
Jul 07 19:40:20.368235 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.368217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"rate-test-model-836c8d-kserve-self-signed-certs\""
Jul 07 19:40:20.377501 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.377481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:40:20.473221 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.473407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.473407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.473407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.473407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.473594 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.473433    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v5m\" (UniqueName: \"kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.573930 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.573893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.573944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.573976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.573999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v5m\" (UniqueName: \"kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.574029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574106 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.574059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574382 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.574335    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574382 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.574343    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.574490 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.574428    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.576302 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.576283    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.576599 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.576578    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.583287 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.583257    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v5m\" (UniqueName: \"kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m\") pod \"rate-test-model-836c8d-kserve-69f7f4659f-lqswh\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") " pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.676752 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.676654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:20.810676 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.810653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:40:20.812799 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:40:20.812761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod792d9ab9_3352_4102_b59b_3e5c744baade.slice/crio-a358df189d3a98c16a067cd6f62f1e73a2759b0891040160f4d7cf20d8438866 WatchSource:0}: Error finding container a358df189d3a98c16a067cd6f62f1e73a2759b0891040160f4d7cf20d8438866: Status 404 returned error can't find the container with id a358df189d3a98c16a067cd6f62f1e73a2759b0891040160f4d7cf20d8438866
Jul 07 19:40:20.849573 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:20.849542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" event={"ID":"792d9ab9-3352-4102-b59b-3e5c744baade","Type":"ContainerStarted","Data":"a358df189d3a98c16a067cd6f62f1e73a2759b0891040160f4d7cf20d8438866"}
Jul 07 19:40:21.855585 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:21.855548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" event={"ID":"792d9ab9-3352-4102-b59b-3e5c744baade","Type":"ContainerStarted","Data":"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"}
Jul 07 19:40:21.876379 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:21.876334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" podStartSLOduration=1.876318569 podStartE2EDuration="1.876318569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:40:21.874390693 +0000 UTC m=+2387.594189885" watchObservedRunningTime="2026-07-07 19:40:21.876318569 +0000 UTC m=+2387.596117759"
Jul 07 19:40:30.677442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:30.677404    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:30.677442 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:30.677449    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:30.690811 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:30.690788    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:40:30.908988 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:40:30.908958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:41:02.465198 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:02.465120    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:41:04.401846 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.401811    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:41:04.402254 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.402034    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerName="maas-api" containerID="cri-o://2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180" gracePeriod=30
Jul 07 19:41:04.518655 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.518601    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.52:8443/health\": dial tcp 10.134.0.52:8443: connect: connection refused"
Jul 07 19:41:04.645619 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.645594    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:41:04.824130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.824102    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") pod \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") "
Jul 07 19:41:04.824306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.824172    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca\") pod \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") "
Jul 07 19:41:04.824306 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.824209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4frh\" (UniqueName: \"kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh\") pod \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\" (UID: \"be73bbd6-d6dc-4841-9029-cc4816fbfb66\") "
Jul 07 19:41:04.824476 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.824452    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be73bbd6-d6dc-4841-9029-cc4816fbfb66" (UID: "be73bbd6-d6dc-4841-9029-cc4816fbfb66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:41:04.826110 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.826085    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be73bbd6-d6dc-4841-9029-cc4816fbfb66" (UID: "be73bbd6-d6dc-4841-9029-cc4816fbfb66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:41:04.826212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.826115    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh" (OuterVolumeSpecName: "kube-api-access-c4frh") pod "be73bbd6-d6dc-4841-9029-cc4816fbfb66" (UID: "be73bbd6-d6dc-4841-9029-cc4816fbfb66"). InnerVolumeSpecName "kube-api-access-c4frh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:41:04.925247 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.925221    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be73bbd6-d6dc-4841-9029-cc4816fbfb66-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:04.925247 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.925245    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be73bbd6-d6dc-4841-9029-cc4816fbfb66-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:04.925403 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:04.925254    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4frh\" (UniqueName: \"kubernetes.io/projected/be73bbd6-d6dc-4841-9029-cc4816fbfb66-kube-api-access-c4frh\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:05.063094 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.063065    2560 generic.go:356] "Generic (PLEG): container finished" podID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerID="2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180" exitCode=0
Jul 07 19:41:05.063219 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.063106    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"
Jul 07 19:41:05.063219 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.063149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" event={"ID":"be73bbd6-d6dc-4841-9029-cc4816fbfb66","Type":"ContainerDied","Data":"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"}
Jul 07 19:41:05.063219 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.063201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb" event={"ID":"be73bbd6-d6dc-4841-9029-cc4816fbfb66","Type":"ContainerDied","Data":"d85b1aa59e69472fe854a7175b4b6af823bd78ac2d850e426ea68329b16a0c57"}
Jul 07 19:41:05.063392 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.063224    2560 scope.go:117] "RemoveContainer" containerID="2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"
Jul 07 19:41:05.072164 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.072147    2560 scope.go:117] "RemoveContainer" containerID="2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"
Jul 07 19:41:05.072405 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:41:05.072385    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180\": container with ID starting with 2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180 not found: ID does not exist" containerID="2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"
Jul 07 19:41:05.072501 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.072411    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180"} err="failed to get container status \"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180\": rpc error: code = NotFound desc = could not find container \"2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180\": container with ID starting with 2da9ed6601780bed8765615aff62e7cc1c1962c7156b2c5015a3650e97e92180 not found: ID does not exist"
Jul 07 19:41:05.090632 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.090581    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:41:05.096397 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.096367    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-0e4903-6cbc757c9d-tmggb"]
Jul 07 19:41:05.408038 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:05.407965    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" path="/var/lib/kubelet/pods/be73bbd6-d6dc-4841-9029-cc4816fbfb66/volumes"
Jul 07 19:41:17.485215 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.485152    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" podUID="792d9ab9-3352-4102-b59b-3e5c744baade" containerName="main" containerID="cri-o://277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19" gracePeriod=45
Jul 07 19:41:17.734859 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.734834    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:41:17.817587 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817558    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6v5m\" (UniqueName: \"kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817852    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home\") pod \"792d9ab9-3352-4102-b59b-3e5c744baade\" (UID: \"792d9ab9-3352-4102-b59b-3e5c744baade\") "
Jul 07 19:41:17.818001 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.817885    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache" (OuterVolumeSpecName: "model-cache") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:17.818304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.818123    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:17.818304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.818159    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home" (OuterVolumeSpecName: "home") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:17.818304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.818245    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-tmp-dir\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:17.818304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.818262    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-home\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:17.818304 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.818276    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-model-cache\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:17.819751 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.819724    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm" (OuterVolumeSpecName: "dshm") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:17.819865 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.819827    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m" (OuterVolumeSpecName: "kube-api-access-n6v5m") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "kube-api-access-n6v5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:41:17.819934 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.819916    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "792d9ab9-3352-4102-b59b-3e5c744baade" (UID: "792d9ab9-3352-4102-b59b-3e5c744baade"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:41:17.919212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.919178    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6v5m\" (UniqueName: \"kubernetes.io/projected/792d9ab9-3352-4102-b59b-3e5c744baade-kube-api-access-n6v5m\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:17.919212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.919207    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/792d9ab9-3352-4102-b59b-3e5c744baade-dshm\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:17.919212 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:17.919216    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792d9ab9-3352-4102-b59b-3e5c744baade-tls-certs\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:18.123407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.123314    2560 generic.go:356] "Generic (PLEG): container finished" podID="792d9ab9-3352-4102-b59b-3e5c744baade" containerID="277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19" exitCode=0
Jul 07 19:41:18.123407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.123363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" event={"ID":"792d9ab9-3352-4102-b59b-3e5c744baade","Type":"ContainerDied","Data":"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"}
Jul 07 19:41:18.123407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.123373    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"
Jul 07 19:41:18.123407 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.123399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh" event={"ID":"792d9ab9-3352-4102-b59b-3e5c744baade","Type":"ContainerDied","Data":"a358df189d3a98c16a067cd6f62f1e73a2759b0891040160f4d7cf20d8438866"}
Jul 07 19:41:18.123714 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.123416    2560 scope.go:117] "RemoveContainer" containerID="277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"
Jul 07 19:41:18.132003 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.131981    2560 scope.go:117] "RemoveContainer" containerID="277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"
Jul 07 19:41:18.132354 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:41:18.132258    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19\": container with ID starting with 277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19 not found: ID does not exist" containerID="277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"
Jul 07 19:41:18.132354 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.132282    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19"} err="failed to get container status \"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19\": rpc error: code = NotFound desc = could not find container \"277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19\": container with ID starting with 277d9c54a64a7530acfca747ded45394f27fc1cfdc75e8131b5a55f669fbff19 not found: ID does not exist"
Jul 07 19:41:18.153579 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.153552    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:41:18.161359 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:18.161341    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/rate-test-model-836c8d-kserve-69f7f4659f-lqswh"]
Jul 07 19:41:19.409910 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:19.409878    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792d9ab9-3352-4102-b59b-3e5c744baade" path="/var/lib/kubelet/pods/792d9ab9-3352-4102-b59b-3e5c744baade/volumes"
Jul 07 19:41:49.250195 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.250159    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:41:49.250801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.250441    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" podUID="6f4d2c01-eee2-47de-bae1-11d50a59700d" containerName="maas-api" containerID="cri-o://91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e" gracePeriod=30
Jul 07 19:41:49.492183 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.492157    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:41:49.548223 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.548162    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca\") pod \"6f4d2c01-eee2-47de-bae1-11d50a59700d\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") "
Jul 07 19:41:49.548351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.548256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls\") pod \"6f4d2c01-eee2-47de-bae1-11d50a59700d\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") "
Jul 07 19:41:49.548351 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.548280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppm4k\" (UniqueName: \"kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k\") pod \"6f4d2c01-eee2-47de-bae1-11d50a59700d\" (UID: \"6f4d2c01-eee2-47de-bae1-11d50a59700d\") "
Jul 07 19:41:49.548484 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.548463    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f4d2c01-eee2-47de-bae1-11d50a59700d" (UID: "6f4d2c01-eee2-47de-bae1-11d50a59700d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:41:49.550294 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.550270    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k" (OuterVolumeSpecName: "kube-api-access-ppm4k") pod "6f4d2c01-eee2-47de-bae1-11d50a59700d" (UID: "6f4d2c01-eee2-47de-bae1-11d50a59700d"). InnerVolumeSpecName "kube-api-access-ppm4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:41:49.550396 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.550340    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f4d2c01-eee2-47de-bae1-11d50a59700d" (UID: "6f4d2c01-eee2-47de-bae1-11d50a59700d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:41:49.649813 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.649786    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d2c01-eee2-47de-bae1-11d50a59700d-openshift-service-ca\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:49.649813 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.649808    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d2c01-eee2-47de-bae1-11d50a59700d-maas-api-tls\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:49.649813 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:49.649820    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppm4k\" (UniqueName: \"kubernetes.io/projected/6f4d2c01-eee2-47de-bae1-11d50a59700d-kube-api-access-ppm4k\") on node \"ip-10-0-140-56.ec2.internal\" DevicePath \"\""
Jul 07 19:41:50.269904 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.269865    2560 generic.go:356] "Generic (PLEG): container finished" podID="6f4d2c01-eee2-47de-bae1-11d50a59700d" containerID="91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e" exitCode=0
Jul 07 19:41:50.270297 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.269910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"
Jul 07 19:41:50.270297 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.269945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" event={"ID":"6f4d2c01-eee2-47de-bae1-11d50a59700d","Type":"ContainerDied","Data":"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"}
Jul 07 19:41:50.270297 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.269998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw" event={"ID":"6f4d2c01-eee2-47de-bae1-11d50a59700d","Type":"ContainerDied","Data":"ddd15401e6d57479fd08d9040b0d9258f92c9f02843e3ccacd2df3c014cc8df7"}
Jul 07 19:41:50.270297 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.270021    2560 scope.go:117] "RemoveContainer" containerID="91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"
Jul 07 19:41:50.278597 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.278575    2560 scope.go:117] "RemoveContainer" containerID="91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"
Jul 07 19:41:50.278878 ip-10-0-140-56 kubenswrapper[2560]: E0707 19:41:50.278860    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e\": container with ID starting with 91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e not found: ID does not exist" containerID="91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"
Jul 07 19:41:50.278937 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.278885    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e"} err="failed to get container status \"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e\": rpc error: code = NotFound desc = could not find container \"91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e\": container with ID starting with 91eb3be8ded435c1e88b361e6ef2d0334631bbec1b1986400a3ed323652efc3e not found: ID does not exist"
Jul 07 19:41:50.295728 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.295686    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:41:50.300410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:50.300380    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-836c8d-79c8b8884b-tplpw"]
Jul 07 19:41:51.407887 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:41:51.407843    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4d2c01-eee2-47de-bae1-11d50a59700d" path="/var/lib/kubelet/pods/6f4d2c01-eee2-47de-bae1-11d50a59700d/volumes"
Jul 07 19:42:02.463380 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:02.463348    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vhk5x_71a5782b-194e-4a47-b998-52ceadacac51/manager/0.log"
Jul 07 19:42:02.593559 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:02.593525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-867c89778c-ct94l_2645e17e-7e3f-4d3d-a191-66cadbeb4b4f/maas-api/0.log"
Jul 07 19:42:02.706901 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:02.706872    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724195-mfx64_1994c35a-0492-4701-a373-97478b35accf/cleanup/0.log"
Jul 07 19:42:02.819834 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:02.819811    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724210-5m9w4_1ed15820-3b62-4529-970f-62c23b63bb4f/cleanup/0.log"
Jul 07 19:42:02.947749 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:02.947717    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57c89485cc-pfqzg_3adf443a-f972-4340-980e-a52ae46ca00a/manager/0.log"
Jul 07 19:42:03.070743 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:03.070660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n7662_42730f4f-2eac-4edf-aad2-73f7ccfc9593/server/0.log"
Jul 07 19:42:03.308254 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:03.308228    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-5rzkk_47f799f5-fba3-4227-8974-258b4d93543d/copy-manifests/0.log"
Jul 07 19:42:03.324128 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:03.324072    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-5rzkk_47f799f5-fba3-4227-8974-258b4d93543d/manager/0.log"
Jul 07 19:42:03.706223 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:03.706155    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-vp45s_47047fc2-fd87-454a-b30f-be0583f1767c/postgres/0.log"
Jul 07 19:42:05.672746 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:05.672716    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ct4vk_bca43827-53bb-4b0e-bef7-83fea49d25e3/manager/0.log"
Jul 07 19:42:06.415342 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:06.415313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tfsk8_301ff9ce-2e49-4b96-8586-642c134645c7/discovery/0.log"
Jul 07 19:42:06.534471 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:06.534446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-2cfpr_3a247b30-5bff-4f74-b342-9bb8cb05215a/kube-auth-proxy/0.log"
Jul 07 19:42:06.920837 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:06.920808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-wgd92_de5be2b7-e91f-4595-9631-01e25e1c81a5/payload-pre-processing/0.log"
Jul 07 19:42:07.040975 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:07.040948    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-bjtzz_6d4ba065-b74b-49c9-b3d8-7e58b070e7d2/payload-processing/0.log"
Jul 07 19:42:07.164180 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:07.164152    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64bcb7459f-kdll9_3cd34889-f57e-46ec-a1aa-6297a9d6c8e5/router/0.log"
Jul 07 19:42:07.646401 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:07.646372    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tswks_6c33c50c-2ad0-405d-beba-245bc30a8ed9/main/0.log"
Jul 07 19:42:07.768764 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:07.768735    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-trfx9_fcfaf158-1bdb-4cd0-84ab-ec0202155685/main/0.log"
Jul 07 19:42:08.008863 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:08.008839    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hhjq6_67ad0a1b-e3d9-4ca9-b8db-544e9a159316/main/0.log"
Jul 07 19:42:15.966327 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:15.966297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-klwmk_443ebe9b-7063-4620-9025-3884c0734c8c/global-pull-secret-syncer/0.log"
Jul 07 19:42:16.276659 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:16.276627    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n54g5_b852f3ea-6aa4-4e1e-82db-92562d7b5fa4/konnectivity-agent/0.log"
Jul 07 19:42:16.310866 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:16.310838    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-56.ec2.internal_e764c2d22d220cd20bf06ff00832f0cd/haproxy/0.log"
Jul 07 19:42:22.003310 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:22.003276    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ct4vk_bca43827-53bb-4b0e-bef7-83fea49d25e3/manager/0.log"
Jul 07 19:42:24.719645 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:24.719614    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbfg8_42f797a1-6b65-4565-87ee-c9ee4f335ef3/node-exporter/0.log"
Jul 07 19:42:24.785638 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:24.785613    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbfg8_42f797a1-6b65-4565-87ee-c9ee4f335ef3/kube-rbac-proxy/0.log"
Jul 07 19:42:24.853964 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:24.853942    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbfg8_42f797a1-6b65-4565-87ee-c9ee4f335ef3/init-textfile/0.log"
Jul 07 19:42:25.598994 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.598963    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"]
Jul 07 19:42:25.599332 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599318    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792d9ab9-3352-4102-b59b-3e5c744baade" containerName="main"
Jul 07 19:42:25.599377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599335    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="792d9ab9-3352-4102-b59b-3e5c744baade" containerName="main"
Jul 07 19:42:25.599377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599350    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerName="maas-api"
Jul 07 19:42:25.599377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599359    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerName="maas-api"
Jul 07 19:42:25.599377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599370    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4d2c01-eee2-47de-bae1-11d50a59700d" containerName="maas-api"
Jul 07 19:42:25.599377 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599376    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4d2c01-eee2-47de-bae1-11d50a59700d" containerName="maas-api"
Jul 07 19:42:25.599542 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599435    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4d2c01-eee2-47de-bae1-11d50a59700d" containerName="maas-api"
Jul 07 19:42:25.599542 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599444    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="792d9ab9-3352-4102-b59b-3e5c744baade" containerName="main"
Jul 07 19:42:25.599542 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.599452    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be73bbd6-d6dc-4841-9029-cc4816fbfb66" containerName="maas-api"
Jul 07 19:42:25.602417 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.602402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.605475 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.605456    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bhqz8\"/\"default-dockercfg-b4nc4\""
Jul 07 19:42:25.605815 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.605798    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"kube-root-ca.crt\""
Jul 07 19:42:25.605865 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.605798    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"openshift-service-ca.crt\""
Jul 07 19:42:25.617824 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.617802    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"]
Jul 07 19:42:25.731784 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.731761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vt7\" (UniqueName: \"kubernetes.io/projected/f6ae4170-d656-49c0-b53a-35cb09e5f581-kube-api-access-j7vt7\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.732079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.731801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-lib-modules\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.732079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.731826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-sys\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.732079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.731883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-proc\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.732079 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.731905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-podres\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832410 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-lib-modules\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-sys\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-proc\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-podres\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832526 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vt7\" (UniqueName: \"kubernetes.io/projected/f6ae4170-d656-49c0-b53a-35cb09e5f581-kube-api-access-j7vt7\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832681 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832519    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-sys\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832681 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832533    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-lib-modules\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832681 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832525    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-proc\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.832681 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.832596    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6ae4170-d656-49c0-b53a-35cb09e5f581-podres\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.842959 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.842931    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vt7\" (UniqueName: \"kubernetes.io/projected/f6ae4170-d656-49c0-b53a-35cb09e5f581-kube-api-access-j7vt7\") pod \"perf-node-gather-daemonset-n7h5q\" (UID: \"f6ae4170-d656-49c0-b53a-35cb09e5f581\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:25.912104 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:25.912050    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:26.043383 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:26.043353    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"]
Jul 07 19:42:26.046073 ip-10-0-140-56 kubenswrapper[2560]: W0707 19:42:26.046045    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf6ae4170_d656_49c0_b53a_35cb09e5f581.slice/crio-18b9d8d6ac099dff550d0b30ec361df637982335055a054ca23d153516a8eeba WatchSource:0}: Error finding container 18b9d8d6ac099dff550d0b30ec361df637982335055a054ca23d153516a8eeba: Status 404 returned error can't find the container with id 18b9d8d6ac099dff550d0b30ec361df637982335055a054ca23d153516a8eeba
Jul 07 19:42:26.442509 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:26.442440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q" event={"ID":"f6ae4170-d656-49c0-b53a-35cb09e5f581","Type":"ContainerStarted","Data":"5f38da6384fe36d0e6c130de16416073744d1ebed994d163eeb1cc5c2be196dd"}
Jul 07 19:42:26.442509 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:26.442500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q" event={"ID":"f6ae4170-d656-49c0-b53a-35cb09e5f581","Type":"ContainerStarted","Data":"18b9d8d6ac099dff550d0b30ec361df637982335055a054ca23d153516a8eeba"}
Jul 07 19:42:26.442736 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:26.442573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:26.470406 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:26.470360    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q" podStartSLOduration=1.470346987 podStartE2EDuration="1.470346987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:26.470168986 +0000 UTC m=+2512.189968176" watchObservedRunningTime="2026-07-07 19:42:26.470346987 +0000 UTC m=+2512.190146177"
Jul 07 19:42:30.494130 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:30.494100    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrz2k_a4393ec0-a42e-4579-a6a6-6bc959d3af3c/dns/0.log"
Jul 07 19:42:30.527337 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:30.527312    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrz2k_a4393ec0-a42e-4579-a6a6-6bc959d3af3c/kube-rbac-proxy/0.log"
Jul 07 19:42:30.566127 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:30.566105    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lp9zc_6c2bd74c-a302-4fd8-94d0-c41ed8537fed/dns-node-resolver/0.log"
Jul 07 19:42:31.299483 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:31.299446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m64kp_23595584-cbfb-4ad5-985a-449f84e148c0/node-ca/0.log"
Jul 07 19:42:32.455696 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.455658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-n7h5q"
Jul 07 19:42:32.654427 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.654391    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tfsk8_301ff9ce-2e49-4b96-8586-642c134645c7/discovery/0.log"
Jul 07 19:42:32.693770 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.693745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-2cfpr_3a247b30-5bff-4f74-b342-9bb8cb05215a/kube-auth-proxy/0.log"
Jul 07 19:42:32.888384 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.888352    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-wgd92_de5be2b7-e91f-4595-9631-01e25e1c81a5/payload-pre-processing/0.log"
Jul 07 19:42:32.948409 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.948383    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-bjtzz_6d4ba065-b74b-49c9-b3d8-7e58b070e7d2/payload-processing/0.log"
Jul 07 19:42:32.977913 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:32.977890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64bcb7459f-kdll9_3cd34889-f57e-46ec-a1aa-6297a9d6c8e5/router/0.log"
Jul 07 19:42:33.889969 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:33.889938    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5l8j2_7a897e22-b61b-4495-b468-047d2f3bc6b5/serve-healthcheck-canary/0.log"
Jul 07 19:42:34.849801 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:34.849721    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrdxq_97fde49a-d038-462a-bcee-cc86d2cec0ec/kube-rbac-proxy/0.log"
Jul 07 19:42:34.885134 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:34.885113    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrdxq_97fde49a-d038-462a-bcee-cc86d2cec0ec/exporter/0.log"
Jul 07 19:42:34.967742 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:34.967716    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qrdxq_97fde49a-d038-462a-bcee-cc86d2cec0ec/extractor/0.log"
Jul 07 19:42:36.512037 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:36.512006    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tfsk8_301ff9ce-2e49-4b96-8586-642c134645c7/discovery/0.log"
Jul 07 19:42:36.539989 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:36.539948    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-2cfpr_3a247b30-5bff-4f74-b342-9bb8cb05215a/kube-auth-proxy/0.log"
Jul 07 19:42:36.750266 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:36.750234    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-wgd92_de5be2b7-e91f-4595-9631-01e25e1c81a5/payload-pre-processing/0.log"
Jul 07 19:42:36.799091 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:36.799011    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-bjtzz_6d4ba065-b74b-49c9-b3d8-7e58b070e7d2/payload-processing/0.log"
Jul 07 19:42:36.821259 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:36.821236    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64bcb7459f-kdll9_3cd34889-f57e-46ec-a1aa-6297a9d6c8e5/router/0.log"
Jul 07 19:42:37.933491 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:37.933461    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vhk5x_71a5782b-194e-4a47-b998-52ceadacac51/manager/0.log"
Jul 07 19:42:37.993890 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:37.993864    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-867c89778c-ct94l_2645e17e-7e3f-4d3d-a191-66cadbeb4b4f/maas-api/0.log"
Jul 07 19:42:38.054161 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.054138    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724195-mfx64_1994c35a-0492-4701-a373-97478b35accf/cleanup/0.log"
Jul 07 19:42:38.106584 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.106561    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724210-5m9w4_1ed15820-3b62-4529-970f-62c23b63bb4f/cleanup/0.log"
Jul 07 19:42:38.270366 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.270328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57c89485cc-pfqzg_3adf443a-f972-4340-980e-a52ae46ca00a/manager/0.log"
Jul 07 19:42:38.281923 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.281896    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tfsk8_301ff9ce-2e49-4b96-8586-642c134645c7/discovery/0.log"
Jul 07 19:42:38.301023 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.301003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-2cfpr_3a247b30-5bff-4f74-b342-9bb8cb05215a/kube-auth-proxy/0.log"
Jul 07 19:42:38.328724 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.328691    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n7662_42730f4f-2eac-4edf-aad2-73f7ccfc9593/server/0.log"
Jul 07 19:42:38.473888 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.473857    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-wgd92_de5be2b7-e91f-4595-9631-01e25e1c81a5/payload-pre-processing/0.log"
Jul 07 19:42:38.548143 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.548078    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-bjtzz_6d4ba065-b74b-49c9-b3d8-7e58b070e7d2/payload-processing/0.log"
Jul 07 19:42:38.584111 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.584089    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64bcb7459f-kdll9_3cd34889-f57e-46ec-a1aa-6297a9d6c8e5/router/0.log"
Jul 07 19:42:38.642767 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.642742    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-5rzkk_47f799f5-fba3-4227-8974-258b4d93543d/manager/0.log"
Jul 07 19:42:38.694887 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.694863    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-5rzkk_47f799f5-fba3-4227-8974-258b4d93543d/copy-manifests/0.log"
Jul 07 19:42:38.996720 ip-10-0-140-56 kubenswrapper[2560]: I0707 19:42:38.996677    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-vp45s_47047fc2-fd87-454a-b30f-be0583f1767c/postgres/0.log"