Jul 31 19:54:28.412274 ip-10-0-143-178 systemd[1]: Starting Kubernetes Kubelet... Jul 31 19:54:28.845495 ip-10-0-143-178 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 19:54:28.845495 ip-10-0-143-178 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 19:54:28.845495 ip-10-0-143-178 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 19:54:28.845495 ip-10-0-143-178 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 19:54:28.845495 ip-10-0-143-178 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 19:54:28.848372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.848284 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 19:54:28.853207 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853194 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853209 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853213 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853216 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853219 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853222 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853225 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853228 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853231 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853233 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853236 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853238 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853241 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853244 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853246 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853248 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853251 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853254 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.853248 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853257 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853259 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853262 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853265 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853268 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853270 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853272 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853275 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853277 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853280 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853282 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853285 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853287 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853289 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853292 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853295 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853297 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853300 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853302 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853304 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.853660 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853307 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853309 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853312 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853314 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853316 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853320 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853322 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853324 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853327 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853329 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853333 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853337 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853340 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853343 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853347 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853350 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853353 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853356 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853358 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853361 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.854143 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853364 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853366 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853369 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853373 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853376 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853379 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853382 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853385 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853387 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853390 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853392 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853394 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853397 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853400 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853402 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853405 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853407 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853410 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853412 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.854611 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853414 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853418 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853421 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853424 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853426 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853429 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853431 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853434 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853436 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853440 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853443 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853446 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853448 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853451 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853453 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853455 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853458 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853460 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853463 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.855070 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853465 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853468 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.853470 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854145 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854152 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854155 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854157 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854160 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854164 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854166 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854169 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854171 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854174 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854176 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854179 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854182 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854184 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854187 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854190 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.855513 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854195 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854201 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854204 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854207 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854210 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854212 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854215 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854218 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854220 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854224 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854228 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854231 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854234 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854237 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854239 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854242 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854244 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854247 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854249 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854251 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.856003 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854254 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854256 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854260 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854262 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854265 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854268 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854270 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854272 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854275 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854278 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854280 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854282 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854285 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854287 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854290 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854292 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854294 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854297 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854299 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854301 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.856477 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854304 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854306 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854309 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854311 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854314 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854317 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854320 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854323 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854325 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854328 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854330 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854333 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854335 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854338 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854340 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854343 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854345 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854347 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854351 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854353 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.856949 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854356 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854358 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854361 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854364 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854366 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854369 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854371 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854373 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854376 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854379 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854381 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854383 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854386 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854388 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854391 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854393 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854396 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854399 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854401 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854403 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.857420 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854406 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854409 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854411 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854746 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854750 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854753 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854755 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854771 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854774 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854776 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854779 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854782 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854784 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854787 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854789 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854792 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854794 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854797 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854799 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854802 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.857914 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854805 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854807 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854810 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854812 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854816 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854820 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854823 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854825 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854828 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854831 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854834 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854836 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854839 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854841 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854844 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854846 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854849 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854853 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.858421 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854856 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854860 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854864 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854866 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854869 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854871 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854874 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854876 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854879 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854881 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854884 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854886 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854889 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854893 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854896 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854898 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854900 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854903 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854905 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.858848 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854908 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854910 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854913 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854915 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854918 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854921 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854923 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854925 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854928 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854930 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854932 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854935 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854937 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854940 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854945 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854947 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854964 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854966 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854969 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854972 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.859295 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854974 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854977 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854979 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854982 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854984 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854987 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854989 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854992 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854994 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854997 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.854999 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855002 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855004 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855007 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855009 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855014 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855017 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855020 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855022 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855025 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.859756 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855027 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855030 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855032 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855035 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855037 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855103 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855111 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855117 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855121 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855126 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855130 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855134 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855137 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855141 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855144 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855147 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855150 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855153 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855156 2556 flags.go:64] FLAG: --cgroup-root="" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855159 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855162 2556 flags.go:64] FLAG: --client-ca-file="" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855165 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855168 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855173 2556 flags.go:64] FLAG: --cluster-domain="" Jul 31 19:54:28.860228 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855175 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855178 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855181 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855186 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855190 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855192 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855196 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855199 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855202 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855205 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855208 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855210 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855213 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855217 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855221 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855224 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855227 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855230 2556 flags.go:64] FLAG: --enable-server="true" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855233 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855237 2556 flags.go:64] FLAG: --event-burst="100" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855241 2556 flags.go:64] FLAG: --event-qps="50" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855243 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855247 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855249 2556 flags.go:64] FLAG: --eviction-hard="" Jul 31 19:54:28.860805 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855253 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855256 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855258 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855261 2556 flags.go:64] FLAG: --eviction-soft="" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855264 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855267 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855269 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855272 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855275 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855278 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855281 2556 flags.go:64] FLAG: --feature-gates="" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855285 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855288 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855291 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855294 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855297 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855299 2556 flags.go:64] FLAG: --help="false" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855302 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-178.ec2.internal" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855305 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855308 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855311 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855314 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855319 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855321 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 19:54:28.861386 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855324 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855327 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855329 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855335 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855338 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855341 2556 flags.go:64] FLAG: --kube-reserved="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855344 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855346 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855350 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855352 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855355 2556 flags.go:64] FLAG: --lock-file="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855358 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855360 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855363 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855368 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855371 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855373 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855376 2556 flags.go:64] FLAG: --logging-format="text" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855379 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855383 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855385 2556 flags.go:64] FLAG: --manifest-url="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855388 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855393 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855396 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855399 2556 flags.go:64] FLAG: --max-pods="110" Jul 31 19:54:28.862002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855402 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855405 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855407 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855410 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855414 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855418 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855420 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855427 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855430 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855432 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855435 2556 flags.go:64] FLAG: --pod-cidr="" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855439 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855443 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855446 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855449 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855452 2556 flags.go:64] FLAG: --port="10250" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855455 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855457 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-070f37344203b1b06" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855460 2556 flags.go:64] FLAG: --qos-reserved="" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855463 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855466 2556 flags.go:64] FLAG: --register-node="true" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855469 2556 flags.go:64] FLAG: --register-with-taints="" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855473 2556 flags.go:64] FLAG: --registry-burst="10" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855475 2556 flags.go:64] FLAG: --registry-qps="5" Jul 31 19:54:28.862655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855478 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855481 2556 flags.go:64] FLAG: --reserved-memory="" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855484 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855488 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855491 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855494 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855496 2556 flags.go:64] FLAG: --runonce="false" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855499 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855502 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855505 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855507 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855510 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855513 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855517 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855520 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855523 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855526 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855529 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855532 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855534 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855539 2556 flags.go:64] FLAG: --system-cgroups="" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855542 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855547 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855550 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855552 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 19:54:28.863324 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855557 2556 flags.go:64] FLAG: --tls-min-version="" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855559 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855562 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855565 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855568 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855571 2556 flags.go:64] FLAG: --v="2" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855574 2556 flags.go:64] FLAG: --version="false" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855578 2556 flags.go:64] FLAG: --vmodule="" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855581 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.855584 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855690 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855694 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855697 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855700 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855703 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855705 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855708 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855711 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855713 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855716 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855719 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.863912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855721 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855724 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855727 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855730 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855732 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855735 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855737 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855740 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855742 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855745 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855747 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855750 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855752 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855754 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855757 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855759 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855762 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855764 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855767 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855769 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.864454 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855772 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855776 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855778 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855781 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855783 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855786 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855788 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855791 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855793 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855796 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855799 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855802 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855804 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855807 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855809 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855812 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855816 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855819 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855822 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855824 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.864932 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855827 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855830 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855833 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855836 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855838 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855841 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855843 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855846 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855848 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855850 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855853 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855856 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855858 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855862 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855864 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855867 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855869 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855872 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855874 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.865441 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855877 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855879 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855882 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855884 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855887 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855889 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855892 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855894 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855896 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855899 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855901 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855904 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855906 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855909 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855913 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855916 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855919 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855922 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855924 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.865875 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855928 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855931 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855934 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855936 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855939 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855941 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855943 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855947 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855962 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.855965 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.856743 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.863667 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.863680 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.863698 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.863704 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 19:54:28.866348 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863757 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863761 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863765 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863767 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863771 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863774 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863776 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863779 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863783 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863785 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863788 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863792 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863797 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863801 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863804 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863806 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863809 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863812 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863814 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863817 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.866704 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863820 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863822 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863825 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863827 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863829 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863832 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863834 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863837 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863840 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863843 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863846 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863850 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863854 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863856 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863859 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863861 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863864 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863866 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863869 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863872 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.867210 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863874 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863876 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863879 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863882 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863884 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863887 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863889 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863891 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863894 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863896 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863899 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863901 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863904 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863906 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863908 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863911 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863913 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863916 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863918 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.867686 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863921 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863924 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863927 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863930 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863933 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863935 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863938 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863940 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863942 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863945 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863948 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863962 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863965 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863969 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863972 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863975 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863978 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863981 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863984 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.868142 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863987 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863990 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863993 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863995 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.863998 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864000 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864003 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864005 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864008 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864010 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864013 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864015 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864018 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864020 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864023 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864026 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864029 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864032 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864034 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864037 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.868595 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864039 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.864044 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864151 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864155 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864159 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864162 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864164 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864167 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864170 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864172 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864175 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864177 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864180 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864183 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864187 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 19:54:28.869130 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864190 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864193 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864195 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864198 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864200 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864203 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864205 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864208 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864210 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864213 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864215 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864217 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864221 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864223 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864226 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864228 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864230 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864233 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864235 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864238 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 19:54:28.869500 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864240 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864242 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864245 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864248 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864250 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864252 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864255 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864257 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864259 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864262 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864264 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864267 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864270 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864272 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864275 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864277 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864280 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864282 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864285 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864287 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 19:54:28.869984 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864290 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864292 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864295 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864297 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864300 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864302 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864306 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864309 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864312 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864315 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864318 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864320 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864323 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864325 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864328 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864330 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864332 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864335 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864337 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 19:54:28.870453 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864340 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864342 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864344 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864347 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864349 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864352 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864354 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864357 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864359 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864362 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864364 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864367 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864369 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864372 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864374 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864377 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864380 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864382 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864384 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864387 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 19:54:28.870890 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864389 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864392 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864394 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864396 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864400 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864403 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:28.864406 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.864411 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.864497 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.866772 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.868557 2556 server.go:1019] "Starting client certificate rotation" Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.868650 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 19:54:28.871372 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.868686 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 19:54:28.871664 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.871417 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 19:54:28.874416 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.874380 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 19:54:28.885674 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.885656 2556 log.go:25] "Validated CRI v1 runtime API" Jul 31 19:54:28.888148 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.888133 2556 log.go:25] "Validated CRI v1 image API" Jul 31 19:54:28.889838 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.889825 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 19:54:28.892441 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.892424 2556 fs.go:135] Filesystem UUIDs: map[51def2c2-342a-4fd6-aaaf-06b1e32a89dd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1508929-37ca-48ec-8a24-8ae1c822e2d9:/dev/nvme0n1p3] Jul 31 19:54:28.892483 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.892441 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 19:54:28.898761 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.898745 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 19:54:28.899503 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.899395 2556 manager.go:216] Machine: {Timestamp:2026-07-31 19:54:28.897906535 +0000 UTC m=+0.359849871 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199961 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2291689c4869aac7a69a2f0adb1df0 SystemUUID:ec229168-9c48-69aa-c7a6-9a2f0adb1df0 BootID:20f78f4d-3620-4e80-b540-b4c2a3b79ada Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:10:48:7f:be:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:10:48:7f:be:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:ad:96:3f:36:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 19:54:28.899503 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.899499 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 19:54:28.899605 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.899576 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 19:54:28.902993 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.902967 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 19:54:28.903132 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.902997 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 19:54:28.903838 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.903829 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 19:54:28.903870 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.903841 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 19:54:28.903870 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.903848 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 19:54:28.904720 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.904710 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 19:54:28.905944 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.905934 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 19:54:28.911085 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911073 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 31 19:54:28.911184 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911175 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 19:54:28.911226 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911204 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 31 19:54:28.911226 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911214 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 19:54:28.911287 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911228 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 19:54:28.911287 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911236 2556 kubelet.go:398] "Adding apiserver pod source" Jul 31 19:54:28.911287 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911244 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 19:54:28.911556 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911544 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 19:54:28.911589 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.911563 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 19:54:28.913803 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.913788 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 19:54:28.915596 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.915580 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 19:54:28.915678 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.915609 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 19:54:28.916995 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.916981 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 19:54:28.917048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917002 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 19:54:28.917048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917012 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 19:54:28.917048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917020 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 19:54:28.917048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917033 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 19:54:28.917048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917041 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917067 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917075 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917085 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917092 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917111 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 19:54:28.917203 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917119 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 19:54:28.917992 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917980 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 19:54:28.918036 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.917993 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 19:54:28.920642 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.920616 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 19:54:28.920642 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.920623 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 19:54:28.920775 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.920656 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 19:54:28.921174 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.921162 2556 server.go:1268] "Started kubelet" Jul 31 19:54:28.921287 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.921246 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 19:54:28.921346 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.921336 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 19:54:28.925834 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.925324 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 19:54:28.925834 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.925685 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 19:54:28.925690 ip-10-0-143-178 systemd[1]: Started Kubernetes Kubelet. Jul 31 19:54:28.927970 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.927938 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 19:54:28.928799 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.928789 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 31 19:54:28.931458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.931440 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xdmfj" Jul 31 19:54:28.932200 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.931049 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-178.ec2.internal.18c777f46aacfeaa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-178.ec2.internal,UID:ip-10-0-143-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-178.ec2.internal,},FirstTimestamp:2026-07-31 19:54:28.921138858 +0000 UTC m=+0.383082193,LastTimestamp:2026-07-31 19:54:28.921138858 +0000 UTC m=+0.383082193,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-178.ec2.internal,}" Jul 31 19:54:28.936357 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.936340 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 19:54:28.936357 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.936353 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 19:54:28.936485 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.936415 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:28.936560 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.936542 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xdmfj" Jul 31 19:54:28.937104 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937083 2556 factory.go:55] Registering systemd factory Jul 31 19:54:28.937217 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937114 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 19:54:28.937217 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937118 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 19:54:28.937217 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937140 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 19:54:28.937217 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937129 2556 factory.go:223] Registration of the systemd container factory successfully Jul 31 19:54:28.937394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937243 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 31 19:54:28.937394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937251 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 31 19:54:28.937394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937341 2556 factory.go:153] Registering CRI-O factory Jul 31 19:54:28.937394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937352 2556 factory.go:223] Registration of the crio container factory successfully Jul 31 19:54:28.937394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937394 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 19:54:28.937573 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937419 2556 factory.go:103] Registering Raw factory Jul 31 19:54:28.937573 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937430 2556 manager.go:1196] Started watching for new ooms in manager Jul 31 19:54:28.937867 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.937851 2556 manager.go:349] Starting recovery of all containers Jul 31 19:54:28.946513 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.946496 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:28.948602 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.948573 2556 manager.go:354] Recovery completed Jul 31 19:54:28.949419 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.949399 2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-178.ec2.internal\" not found" node="ip-10-0-143-178.ec2.internal" Jul 31 19:54:28.953208 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.953197 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:28.955014 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.954973 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:28.955014 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955005 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:28.955121 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955018 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:28.955579 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955563 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 19:54:28.955579 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955577 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:28.955696 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955578 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 19:54:28.955696 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.955647 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 31 19:54:28.956269 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.956256 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:28.956321 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.956278 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:28.956321 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.956287 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:28.957624 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.957612 2556 policy_none.go:49] "None policy: Start" Jul 31 19:54:28.957676 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.957627 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 19:54:28.957676 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.957637 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 19:54:28.960540 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.960527 2556 policy_none.go:47] "Start" Jul 31 19:54:28.987644 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.987628 2556 manager.go:337] "Starting Device Plugin manager" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.987672 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.987685 2556 server.go:87] "Starting device plugin registration server" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.987900 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.987912 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.988306 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.988380 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:28.988389 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.988867 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 19:54:29.015465 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:28.988917 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.038317 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.038293 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 19:54:29.039414 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.039395 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 19:54:29.039499 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.039429 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 19:54:29.039499 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.039449 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 19:54:29.039499 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.039478 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 19:54:29.041994 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.041970 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:29.088208 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.088185 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:29.089268 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.089252 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:29.089363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.089288 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:29.089363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.089302 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:29.089363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.089332 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.097405 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.097365 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.097405 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.097384 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-178.ec2.internal\": node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.111029 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.111013 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.139760 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.139743 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal"] Jul 31 19:54:29.139848 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.139795 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:29.140592 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.140577 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:29.140647 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.140609 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:29.140647 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.140623 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:29.142944 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.142931 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:29.143044 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143031 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.143086 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143057 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:29.143653 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143636 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:29.143653 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143648 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:29.143754 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143668 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:29.143754 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143678 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:29.143754 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143668 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:29.143754 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.143730 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:29.145914 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.145899 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.146011 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.145930 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 19:54:29.146560 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.146546 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientMemory" Jul 31 19:54:29.146626 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.146579 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 19:54:29.146626 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.146589 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeHasSufficientPID" Jul 31 19:54:29.159462 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.159445 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-178.ec2.internal\" not found" node="ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.163703 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.163689 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-178.ec2.internal\" not found" node="ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.212042 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.212024 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.239453 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.239438 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.239560 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.239460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2de2a3e1acdc244afa40a51cd31b1009-config\") pod \"kube-apiserver-proxy-ip-10-0-143-178.ec2.internal\" (UID: \"2de2a3e1acdc244afa40a51cd31b1009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.239560 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.239479 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.312272 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.312248 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.340588 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.340634 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.340634 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340611 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2de2a3e1acdc244afa40a51cd31b1009-config\") pod \"kube-apiserver-proxy-ip-10-0-143-178.ec2.internal\" (UID: \"2de2a3e1acdc244afa40a51cd31b1009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.340701 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340667 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.340732 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2de2a3e1acdc244afa40a51cd31b1009-config\") pod \"kube-apiserver-proxy-ip-10-0-143-178.ec2.internal\" (UID: \"2de2a3e1acdc244afa40a51cd31b1009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.340763 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.340742 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d621a93826c1936883cd721a37b95a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal\" (UID: \"d7d621a93826c1936883cd721a37b95a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.412843 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.412788 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.461363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.461339 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.466945 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:29.466720 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:29.513429 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.513399 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.613864 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.613838 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:29.714425 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:29.714356 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:31.935195 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:31.935172 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:31.987671 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:31.987643 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 19:54:31.988045 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:31.987799 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 19:54:31.988045 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:31.987818 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 19:54:32.036161 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:32.036135 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:32.063684 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.063641 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 19:49:31 +0000 UTC" deadline="2028-03-29 19:06:48.075016311 +0000 UTC" Jul 31 19:54:32.063684 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.063683 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14567h12m16.011338392s" Jul 31 19:54:32.128912 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:32.128877 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d621a93826c1936883cd721a37b95a.slice/crio-f3ac5dcb38b25baeed3468e6fe1cfb75c1513bf08b2f429ae0eb98b1d195bd51 WatchSource:0}: Error finding container f3ac5dcb38b25baeed3468e6fe1cfb75c1513bf08b2f429ae0eb98b1d195bd51: Status 404 returned error can't find the container with id f3ac5dcb38b25baeed3468e6fe1cfb75c1513bf08b2f429ae0eb98b1d195bd51 Jul 31 19:54:32.129136 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:32.129113 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de2a3e1acdc244afa40a51cd31b1009.slice/crio-e4e1a45f108d8a00c56807bd7806047a55875d64f96b6426bbc0d596ca640009 WatchSource:0}: Error finding container e4e1a45f108d8a00c56807bd7806047a55875d64f96b6426bbc0d596ca640009: Status 404 returned error can't find the container with id e4e1a45f108d8a00c56807bd7806047a55875d64f96b6426bbc0d596ca640009 Jul 31 19:54:32.133212 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.133197 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:54:32.136831 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:32.136815 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:32.162566 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.162530 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" event={"ID":"d7d621a93826c1936883cd721a37b95a","Type":"ContainerStarted","Data":"f3ac5dcb38b25baeed3468e6fe1cfb75c1513bf08b2f429ae0eb98b1d195bd51"} Jul 31 19:54:32.163424 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.163400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" event={"ID":"2de2a3e1acdc244afa40a51cd31b1009","Type":"ContainerStarted","Data":"e4e1a45f108d8a00c56807bd7806047a55875d64f96b6426bbc0d596ca640009"} Jul 31 19:54:32.237656 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:32.237632 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:32.338132 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:32.338091 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:32.438694 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:32.438622 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-178.ec2.internal\" not found" Jul 31 19:54:32.503670 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.503645 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:32.556296 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.556276 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:32.557747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.557731 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" Jul 31 19:54:32.567664 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.567648 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 19:54:32.568688 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.568663 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" Jul 31 19:54:32.576541 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:32.576526 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 19:54:33.022716 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.022684 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:33.032661 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.032640 2556 apiserver.go:52] "Watching apiserver" Jul 31 19:54:33.038857 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.038827 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 19:54:33.041299 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.040596 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-wncgg","kube-system/global-pull-secret-syncer-hf82d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5","openshift-cluster-node-tuning-operator/tuned-sfl2b","openshift-dns/node-resolver-c2l2b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal","openshift-multus/multus-additional-cni-plugins-k6h2z","openshift-multus/multus-fn5g2","openshift-multus/network-metrics-daemon-qn4bd","kube-system/konnectivity-agent-c9t5t","kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal","openshift-image-registry/node-ca-h5d27","openshift-network-diagnostics/network-check-target-tncq5","openshift-network-operator/iptables-alerter-hcg55"] Jul 31 19:54:33.043462 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.043443 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.045756 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.045736 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:33.045861 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.045796 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:33.045861 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.045844 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 19:54:33.045976 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.045941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 19:54:33.046023 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.045997 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.046077 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.046060 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 19:54:33.046359 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.046340 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f95tk\"" Jul 31 19:54:33.046787 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.046769 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.048045 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.048029 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.050128 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.050103 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.050502 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.050473 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 19:54:33.050592 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.050521 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w9dfs\"" Jul 31 19:54:33.050592 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.050577 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.050700 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.050619 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.052751 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.052170 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.052751 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.052413 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9gp2h\"" Jul 31 19:54:33.052751 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.052645 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.053358 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.052806 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.054888 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.054848 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.054888 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.054865 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kjwq8\"" Jul 31 19:54:33.055035 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.054941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.055387 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.055373 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.058288 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058190 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qc4bc\"" Jul 31 19:54:33.058288 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058217 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.058444 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058333 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.058444 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058339 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 19:54:33.058444 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.058410 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:33.058444 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058444 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 19:54:33.058679 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.058665 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.061658 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.061640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.064354 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.064333 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 19:54:33.064816 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.064780 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 19:54:33.064909 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.064823 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 19:54:33.064909 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.064846 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.066011 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.065871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 19:54:33.066011 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.065878 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdkqj\"" Jul 31 19:54:33.066486 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.066464 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.066571 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.066548 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:33.066778 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.066757 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.067851 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.067788 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.068747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068557 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-registration-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.068747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068599 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09eabf70-c174-4ddd-958c-8b43daed39dd-hosts-file\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.068747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltl2r\" (UniqueName: \"kubernetes.io/projected/4f5f107e-99d4-4459-82b5-49dcdb626678-kube-api-access-ltl2r\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.068747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068648 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5m6\" (UniqueName: \"kubernetes.io/projected/7a481939-8e1f-4acd-9361-e7095e8d52f3-kube-api-access-fw5m6\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysconfig\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-sys-fs\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068843 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-modprobe-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-conf\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cnibin\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.068990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069018 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-device-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.069109 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069078 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-var-lib-kubelet\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-tuned\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f5f107e-99d4-4459-82b5-49dcdb626678-iptables-alerter-script\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069184 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-systemd\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-run\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-lib-modules\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069279 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjms\" (UniqueName: \"kubernetes.io/projected/716ba6f1-ffa5-4f25-94d6-d4931699f301-kube-api-access-scjms\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09eabf70-c174-4ddd-958c-8b43daed39dd-tmp-dir\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-system-cni-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069425 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crf48\" (UniqueName: \"kubernetes.io/projected/09eabf70-c174-4ddd-958c-8b43daed39dd-kube-api-access-crf48\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f5f107e-99d4-4459-82b5-49dcdb626678-host-slash\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.069582 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069549 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069597 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-os-release\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-socket-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069654 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-host\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069683 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr48\" (UniqueName: \"kubernetes.io/projected/e4e72d7e-54cd-404c-817c-eb5baa46fe50-kube-api-access-cgr48\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069708 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xc5\" (UniqueName: \"kubernetes.io/projected/1b7466ef-1f28-432c-bc6d-93b443cc01db-kube-api-access-62xc5\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-kubernetes\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069852 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ftbzt\"" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.069877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-sys\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.070069 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 19:54:33.070159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.070090 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 19:54:33.071435 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.071359 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.071435 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.071425 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 19:54:33.072012 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.071888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-tmp\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.072012 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.071976 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.072159 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.072023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.073979 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.073930 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vlw6j\"" Jul 31 19:54:33.074195 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.074177 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 19:54:33.075623 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.075601 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.077572 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.077554 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bjjdg\"" Jul 31 19:54:33.077822 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.077807 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 19:54:33.077901 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.077873 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 19:54:33.077960 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.077817 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 19:54:33.088458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.088437 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-49mgl" Jul 31 19:54:33.097303 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.097286 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-49mgl" Jul 31 19:54:33.118544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.118523 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 19:54:33.158251 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.158227 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 19:54:33.172418 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.172418 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172404 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b788b4e-4803-4515-bba2-39b870e004f7-agent-certs\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-socket-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172512 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29qs\" (UniqueName: \"kubernetes.io/projected/52ae78ac-54e5-49b8-8447-8f7f17fd318a-kube-api-access-n29qs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xc5\" (UniqueName: \"kubernetes.io/projected/1b7466ef-1f28-432c-bc6d-93b443cc01db-kube-api-access-62xc5\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b788b4e-4803-4515-bba2-39b870e004f7-konnectivity-ca\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172589 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.172628 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172616 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172672 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172695 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-bin\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172718 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-multus\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysconfig\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172764 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172787 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-etc-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172822 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-kubelet\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-hostroot\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-conf\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172890 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-systemd-units\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172924 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-env-overrides\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.172987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.172951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-netns\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-conf\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173193 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cnibin\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysconfig\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173367 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-socket-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-sysctl-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173468 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-var-lib-kubelet\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-var-lib-kubelet\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173532 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cnibin\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-netd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.173618 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173626 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0af497c1-b530-46e6-861d-8324d9d3d829-host\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173668 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.173675 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.173688 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:33.673668795 +0000 UTC m=+3.015252423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0af497c1-b530-46e6-861d-8324d9d3d829-serviceca\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173779 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.173800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-systemd\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-systemd\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174246 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174341 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4e72d7e-54cd-404c-817c-eb5baa46fe50-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-run\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-lib-modules\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174415 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-systemd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-node-log\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntdk\" (UniqueName: \"kubernetes.io/projected/806b7291-7bec-44fd-942d-8e0ec25d7ab8-kube-api-access-fntdk\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174492 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-run\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-ovn\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.174544 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-lib-modules\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174552 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-script-lib\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174574 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174579 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-os-release\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174607 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-bin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-etc-kubernetes\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174665 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crf48\" (UniqueName: \"kubernetes.io/projected/09eabf70-c174-4ddd-958c-8b43daed39dd-kube-api-access-crf48\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f5f107e-99d4-4459-82b5-49dcdb626678-host-slash\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-os-release\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-host\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-kubelet\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174773 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-netns\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174780 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f5f107e-99d4-4459-82b5-49dcdb626678-host-slash\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174794 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-log-socket\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174836 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-host\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174837 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-os-release\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovn-node-metrics-cert\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175161 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr48\" (UniqueName: \"kubernetes.io/projected/e4e72d7e-54cd-404c-817c-eb5baa46fe50-kube-api-access-cgr48\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174959 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-kubernetes\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.174997 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-sys\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-tmp\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-kubernetes\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-slash\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175070 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-config\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175071 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-sys\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdp8c\" (UniqueName: \"kubernetes.io/projected/0af497c1-b530-46e6-861d-8324d9d3d829-kube-api-access-jdp8c\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175260 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-registration-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175304 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-kubelet-config\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175371 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-registration-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cnibin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175375 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-k8s-cni-cncf-io\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-daemon-config\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.175932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09eabf70-c174-4ddd-958c-8b43daed39dd-hosts-file\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltl2r\" (UniqueName: \"kubernetes.io/projected/4f5f107e-99d4-4459-82b5-49dcdb626678-kube-api-access-ltl2r\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175557 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09eabf70-c174-4ddd-958c-8b43daed39dd-hosts-file\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5m6\" (UniqueName: \"kubernetes.io/projected/7a481939-8e1f-4acd-9361-e7095e8d52f3-kube-api-access-fw5m6\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-conf-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-sys-fs\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-modprobe-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175729 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-modprobe-d\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-sys-fs\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-var-lib-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-dbus\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-system-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175932 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-multus-certs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-device-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.175982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-tuned\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176008 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f5f107e-99d4-4459-82b5-49dcdb626678-iptables-alerter-script\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjms\" (UniqueName: \"kubernetes.io/projected/716ba6f1-ffa5-4f25-94d6-d4931699f301-kube-api-access-scjms\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.176814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b7466ef-1f28-432c-bc6d-93b443cc01db-device-dir\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cni-binary-copy\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176126 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-socket-dir-parent\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09eabf70-c174-4ddd-958c-8b43daed39dd-tmp-dir\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-system-cni-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176385 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4e72d7e-54cd-404c-817c-eb5baa46fe50-system-cni-dir\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176604 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09eabf70-c174-4ddd-958c-8b43daed39dd-tmp-dir\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.177576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.176984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f5f107e-99d4-4459-82b5-49dcdb626678-iptables-alerter-script\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.178723 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.178704 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:33.178818 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.178728 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:33.178818 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.178741 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:33.178818 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.178796 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:33.678780459 +0000 UTC m=+3.020364083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:33.178971 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.178846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-tmp\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.178971 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.178884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/716ba6f1-ffa5-4f25-94d6-d4931699f301-etc-tuned\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.181351 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.181255 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xc5\" (UniqueName: \"kubernetes.io/projected/1b7466ef-1f28-432c-bc6d-93b443cc01db-kube-api-access-62xc5\") pod \"aws-ebs-csi-driver-node-8nfz5\" (UID: \"1b7466ef-1f28-432c-bc6d-93b443cc01db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.183645 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.183620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5m6\" (UniqueName: \"kubernetes.io/projected/7a481939-8e1f-4acd-9361-e7095e8d52f3-kube-api-access-fw5m6\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.183830 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.183815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crf48\" (UniqueName: \"kubernetes.io/projected/09eabf70-c174-4ddd-958c-8b43daed39dd-kube-api-access-crf48\") pod \"node-resolver-c2l2b\" (UID: \"09eabf70-c174-4ddd-958c-8b43daed39dd\") " pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.183930 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.183900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjms\" (UniqueName: \"kubernetes.io/projected/716ba6f1-ffa5-4f25-94d6-d4931699f301-kube-api-access-scjms\") pod \"tuned-sfl2b\" (UID: \"716ba6f1-ffa5-4f25-94d6-d4931699f301\") " pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.184001 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.183981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr48\" (UniqueName: \"kubernetes.io/projected/e4e72d7e-54cd-404c-817c-eb5baa46fe50-kube-api-access-cgr48\") pod \"multus-additional-cni-plugins-k6h2z\" (UID: \"e4e72d7e-54cd-404c-817c-eb5baa46fe50\") " pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.184105 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.184079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltl2r\" (UniqueName: \"kubernetes.io/projected/4f5f107e-99d4-4459-82b5-49dcdb626678-kube-api-access-ltl2r\") pod \"iptables-alerter-hcg55\" (UID: \"4f5f107e-99d4-4459-82b5-49dcdb626678\") " pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.277381 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29qs\" (UniqueName: \"kubernetes.io/projected/52ae78ac-54e5-49b8-8447-8f7f17fd318a-kube-api-access-n29qs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277381 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277341 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b788b4e-4803-4515-bba2-39b870e004f7-konnectivity-ca\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.277381 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277409 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-bin\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-multus\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-etc-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-kubelet\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277489 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-hostroot\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-systemd-units\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277522 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-bin\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277567 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-env-overrides\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277582 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-multus\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.277608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277576 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-etc-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-kubelet\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277622 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277655 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-hostroot\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-systemd-units\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-netns\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277763 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-netd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0af497c1-b530-46e6-861d-8324d9d3d829-host\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0af497c1-b530-46e6-861d-8324d9d3d829-serviceca\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277812 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-netns\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277831 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-systemd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277835 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-cni-netd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277845 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0af497c1-b530-46e6-861d-8324d9d3d829-host\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-node-log\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277882 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-systemd\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277877 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntdk\" (UniqueName: \"kubernetes.io/projected/806b7291-7bec-44fd-942d-8e0ec25d7ab8-kube-api-access-fntdk\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277885 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b788b4e-4803-4515-bba2-39b870e004f7-konnectivity-ca\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.277958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.278153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278003 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-env-overrides\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278006 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-ovn\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-node-log\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278053 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-script-lib\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.278075 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-os-release\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278104 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-bin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.278127 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:54:33.778112461 +0000 UTC m=+3.119696069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278139 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-var-lib-cni-bin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278153 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-etc-kubernetes\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-os-release\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-kubelet\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278197 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0af497c1-b530-46e6-861d-8324d9d3d829-serviceca\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-netns\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-log-socket\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278239 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-kubelet\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovn-node-metrics-cert\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-etc-kubernetes\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279008 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-slash\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278350 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-config\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-run-netns\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-ovn\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278373 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdp8c\" (UniqueName: \"kubernetes.io/projected/0af497c1-b530-46e6-861d-8324d9d3d829-kube-api-access-jdp8c\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278395 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-kubelet-config\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-slash\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cnibin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-log-socket\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-k8s-cni-cncf-io\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-daemon-config\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278463 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-k8s-cni-cncf-io\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-conf-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-var-lib-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-run-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278524 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-dbus\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-system-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.279832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-multus-certs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cni-binary-copy\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-socket-dir-parent\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b788b4e-4803-4515-bba2-39b870e004f7-agent-certs\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-script-lib\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278799 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-system-cni-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278802 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-conf-dir\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-dbus\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278905 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cnibin\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278904 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-kubelet-config\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278942 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-host-run-multus-certs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovnkube-config\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.278980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-socket-dir-parent\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.279022 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.279024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/806b7291-7bec-44fd-942d-8e0ec25d7ab8-var-lib-openvswitch\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.279258 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-multus-daemon-config\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.280642 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.279386 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52ae78ac-54e5-49b8-8447-8f7f17fd318a-cni-binary-copy\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.281332 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.281276 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/806b7291-7bec-44fd-942d-8e0ec25d7ab8-ovn-node-metrics-cert\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.281391 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.281352 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b788b4e-4803-4515-bba2-39b870e004f7-agent-certs\") pod \"konnectivity-agent-c9t5t\" (UID: \"4b788b4e-4803-4515-bba2-39b870e004f7\") " pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.288819 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.288796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntdk\" (UniqueName: \"kubernetes.io/projected/806b7291-7bec-44fd-942d-8e0ec25d7ab8-kube-api-access-fntdk\") pod \"ovnkube-node-wncgg\" (UID: \"806b7291-7bec-44fd-942d-8e0ec25d7ab8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.289197 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.289179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdp8c\" (UniqueName: \"kubernetes.io/projected/0af497c1-b530-46e6-861d-8324d9d3d829-kube-api-access-jdp8c\") pod \"node-ca-h5d27\" (UID: \"0af497c1-b530-46e6-861d-8324d9d3d829\") " pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.289197 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.289193 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29qs\" (UniqueName: \"kubernetes.io/projected/52ae78ac-54e5-49b8-8447-8f7f17fd318a-kube-api-access-n29qs\") pod \"multus-fn5g2\" (UID: \"52ae78ac-54e5-49b8-8447-8f7f17fd318a\") " pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.357100 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.357077 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" Jul 31 19:54:33.364173 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.364154 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" Jul 31 19:54:33.364472 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.364448 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e72d7e_54cd_404c_817c_eb5baa46fe50.slice/crio-c6e7b6fe4c566b9f3f0d55a0d04f8e56d9c746eb916c6b6f9181a41f28b2c9de WatchSource:0}: Error finding container c6e7b6fe4c566b9f3f0d55a0d04f8e56d9c746eb916c6b6f9181a41f28b2c9de: Status 404 returned error can't find the container with id c6e7b6fe4c566b9f3f0d55a0d04f8e56d9c746eb916c6b6f9181a41f28b2c9de Jul 31 19:54:33.371763 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.371742 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7466ef_1f28_432c_bc6d_93b443cc01db.slice/crio-facece93ee9a52f2200fffa2abbbc47671e2ec7505b8665f36061d0b26057537 WatchSource:0}: Error finding container facece93ee9a52f2200fffa2abbbc47671e2ec7505b8665f36061d0b26057537: Status 404 returned error can't find the container with id facece93ee9a52f2200fffa2abbbc47671e2ec7505b8665f36061d0b26057537 Jul 31 19:54:33.376038 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.376018 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" Jul 31 19:54:33.382502 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.382487 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c2l2b" Jul 31 19:54:33.389875 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.389861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcg55" Jul 31 19:54:33.396420 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.396402 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:33.401525 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.401501 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716ba6f1_ffa5_4f25_94d6_d4931699f301.slice/crio-5c16d17939daf98fd45cfcc259ebcf24fcc88e899fbbaaa882cd48e1740ca3a1 WatchSource:0}: Error finding container 5c16d17939daf98fd45cfcc259ebcf24fcc88e899fbbaaa882cd48e1740ca3a1: Status 404 returned error can't find the container with id 5c16d17939daf98fd45cfcc259ebcf24fcc88e899fbbaaa882cd48e1740ca3a1 Jul 31 19:54:33.402141 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.402123 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eabf70_c174_4ddd_958c_8b43daed39dd.slice/crio-2434aef2a355093198feba5054f82b368c50988a8a0265db3f758739e0bb6c0a WatchSource:0}: Error finding container 2434aef2a355093198feba5054f82b368c50988a8a0265db3f758739e0bb6c0a: Status 404 returned error can't find the container with id 2434aef2a355093198feba5054f82b368c50988a8a0265db3f758739e0bb6c0a Jul 31 19:54:33.402992 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.402974 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:33.403656 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.403633 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5f107e_99d4_4459_82b5_49dcdb626678.slice/crio-81c2935069a7b00d2429bbfec07a4dc226f308467d31a60cd434f002983230c3 WatchSource:0}: Error finding container 81c2935069a7b00d2429bbfec07a4dc226f308467d31a60cd434f002983230c3: Status 404 returned error can't find the container with id 81c2935069a7b00d2429bbfec07a4dc226f308467d31a60cd434f002983230c3 Jul 31 19:54:33.408264 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.408242 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806b7291_7bec_44fd_942d_8e0ec25d7ab8.slice/crio-282ccd2040471dcdd4854fb82c50c848cd969e5afeb5ccd5ea2a5918d0dde8d9 WatchSource:0}: Error finding container 282ccd2040471dcdd4854fb82c50c848cd969e5afeb5ccd5ea2a5918d0dde8d9: Status 404 returned error can't find the container with id 282ccd2040471dcdd4854fb82c50c848cd969e5afeb5ccd5ea2a5918d0dde8d9 Jul 31 19:54:33.410849 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.410208 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fn5g2" Jul 31 19:54:33.414988 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.414966 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5d27" Jul 31 19:54:33.415663 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.415621 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b788b4e_4803_4515_bba2_39b870e004f7.slice/crio-e8a079848c8ad1d46cb76e0b6e03f84bc12eea52169e35698a821215fcf7849a WatchSource:0}: Error finding container e8a079848c8ad1d46cb76e0b6e03f84bc12eea52169e35698a821215fcf7849a: Status 404 returned error can't find the container with id e8a079848c8ad1d46cb76e0b6e03f84bc12eea52169e35698a821215fcf7849a Jul 31 19:54:33.420414 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.420386 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ae78ac_54e5_49b8_8447_8f7f17fd318a.slice/crio-5e8ad1ec346f36b2868dfde2272afd30d2fb309742fb2d76b7fc780ca7b972a5 WatchSource:0}: Error finding container 5e8ad1ec346f36b2868dfde2272afd30d2fb309742fb2d76b7fc780ca7b972a5: Status 404 returned error can't find the container with id 5e8ad1ec346f36b2868dfde2272afd30d2fb309742fb2d76b7fc780ca7b972a5 Jul 31 19:54:33.426123 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:54:33.426103 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af497c1_b530_46e6_861d_8324d9d3d829.slice/crio-9d5a7d2d4b127bb790f2e18fccf2dcb5952d55e0225e08fc4987026f4827f9cb WatchSource:0}: Error finding container 9d5a7d2d4b127bb790f2e18fccf2dcb5952d55e0225e08fc4987026f4827f9cb: Status 404 returned error can't find the container with id 9d5a7d2d4b127bb790f2e18fccf2dcb5952d55e0225e08fc4987026f4827f9cb Jul 31 19:54:33.681293 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.681263 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:33.681469 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681447 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:33.681524 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681480 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:33.681524 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.681424 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:33.681524 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681494 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:33.681632 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681534 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:33.681632 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681559 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:34.681540078 +0000 UTC m=+4.023123685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:33.681632 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.681584 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:34.681567628 +0000 UTC m=+4.023151250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:33.782336 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:33.782290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:33.782503 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.782487 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:33.782631 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:33.782563 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:54:34.782532524 +0000 UTC m=+4.124116143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:34.100468 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.100365 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 19:49:33 +0000 UTC" deadline="2028-04-07 04:02:59.4271249 +0000 UTC" Jul 31 19:54:34.100468 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.100421 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14768h8m25.326708558s" Jul 31 19:54:34.160931 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.160900 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:34.161084 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.161030 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:34.194700 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.194640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerStarted","Data":"c6e7b6fe4c566b9f3f0d55a0d04f8e56d9c746eb916c6b6f9181a41f28b2c9de"} Jul 31 19:54:34.211055 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.210949 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5d27" event={"ID":"0af497c1-b530-46e6-861d-8324d9d3d829","Type":"ContainerStarted","Data":"9d5a7d2d4b127bb790f2e18fccf2dcb5952d55e0225e08fc4987026f4827f9cb"} Jul 31 19:54:34.227759 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.227724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fn5g2" event={"ID":"52ae78ac-54e5-49b8-8447-8f7f17fd318a","Type":"ContainerStarted","Data":"5e8ad1ec346f36b2868dfde2272afd30d2fb309742fb2d76b7fc780ca7b972a5"} Jul 31 19:54:34.235916 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.235882 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9t5t" event={"ID":"4b788b4e-4803-4515-bba2-39b870e004f7","Type":"ContainerStarted","Data":"e8a079848c8ad1d46cb76e0b6e03f84bc12eea52169e35698a821215fcf7849a"} Jul 31 19:54:34.242567 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.242537 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcg55" event={"ID":"4f5f107e-99d4-4459-82b5-49dcdb626678","Type":"ContainerStarted","Data":"81c2935069a7b00d2429bbfec07a4dc226f308467d31a60cd434f002983230c3"} Jul 31 19:54:34.260728 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.260662 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" event={"ID":"716ba6f1-ffa5-4f25-94d6-d4931699f301","Type":"ContainerStarted","Data":"5c16d17939daf98fd45cfcc259ebcf24fcc88e899fbbaaa882cd48e1740ca3a1"} Jul 31 19:54:34.268166 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.266198 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" event={"ID":"1b7466ef-1f28-432c-bc6d-93b443cc01db","Type":"ContainerStarted","Data":"facece93ee9a52f2200fffa2abbbc47671e2ec7505b8665f36061d0b26057537"} Jul 31 19:54:34.278519 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.278490 2556 generic.go:356] "Generic (PLEG): container finished" podID="d7d621a93826c1936883cd721a37b95a" containerID="1ad35b61e54eedbf3c2a5c2f29f19cdd9e77863c5b64a0ff5dbcee781c03266a" exitCode=0 Jul 31 19:54:34.278828 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.278799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" event={"ID":"d7d621a93826c1936883cd721a37b95a","Type":"ContainerDied","Data":"1ad35b61e54eedbf3c2a5c2f29f19cdd9e77863c5b64a0ff5dbcee781c03266a"} Jul 31 19:54:34.293584 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.293558 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"282ccd2040471dcdd4854fb82c50c848cd969e5afeb5ccd5ea2a5918d0dde8d9"} Jul 31 19:54:34.311001 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.310977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c2l2b" event={"ID":"09eabf70-c174-4ddd-958c-8b43daed39dd","Type":"ContainerStarted","Data":"2434aef2a355093198feba5054f82b368c50988a8a0265db3f758739e0bb6c0a"} Jul 31 19:54:34.690683 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.690322 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:34.690840 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.690695 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:34.690894 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.690864 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:34.690939 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.690930 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:36.690909867 +0000 UTC m=+6.032493492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:34.691405 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.691388 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:34.691507 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.691410 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:34.691507 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.691423 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:34.691507 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.691468 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:36.691452559 +0000 UTC m=+6.033036183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:34.792079 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:34.792041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:34.792254 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.792191 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:34.792254 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:34.792250 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:54:36.79223298 +0000 UTC m=+6.133816598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:35.101469 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:35.101364 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 19:49:33 +0000 UTC" deadline="2027-12-26 15:42:38.549803863 +0000 UTC" Jul 31 19:54:35.101469 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:35.101424 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12307h48m3.448383869s" Jul 31 19:54:35.163894 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:35.163864 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:35.164063 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:35.164000 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:35.164428 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:35.164398 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:35.164529 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:35.164505 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:35.349886 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:35.349850 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" event={"ID":"d7d621a93826c1936883cd721a37b95a","Type":"ContainerStarted","Data":"11c913512434ee15c8c5ec99d1224a83e6378f30976697837e98dbc14e77eb03"} Jul 31 19:54:36.160996 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:36.160965 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:36.161588 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.161091 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:36.709461 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:36.709403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:36.709657 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:36.709508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:36.709657 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.709642 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:36.709785 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.709704 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:40.709686945 +0000 UTC m=+10.051270550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:36.710035 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.710011 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:36.710116 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.710042 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:36.710116 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.710055 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:36.710116 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.710111 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:40.710093911 +0000 UTC m=+10.051677517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:36.810426 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:36.810296 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:36.810577 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.810429 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:36.810577 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:36.810502 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:54:40.810483179 +0000 UTC m=+10.152066802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:37.160982 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:37.160901 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:37.161138 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:37.161051 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:37.161138 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:37.161131 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:37.161553 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:37.161227 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:38.160825 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:38.160788 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:38.161011 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:38.160929 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:39.160610 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:39.160577 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:39.161039 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:39.160700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:39.161039 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:39.160705 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:39.161039 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:39.160790 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:40.160823 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:40.160672 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:40.160823 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.160799 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:40.743065 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:40.743023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:40.743230 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:40.743071 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:40.743230 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743187 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:40.743230 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743211 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:40.743230 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743231 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:40.743451 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743242 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:40.743451 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743254 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:48.743233711 +0000 UTC m=+18.084817328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:40.743451 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.743289 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:54:48.743273157 +0000 UTC m=+18.084856765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:40.843679 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:40.843643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:40.843857 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.843801 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:40.843920 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:40.843874 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:54:48.843852579 +0000 UTC m=+18.185436220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:41.162352 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:41.161856 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:41.162352 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:41.161970 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:41.162352 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:41.162045 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:41.162352 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:41.162145 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:42.161124 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:42.160972 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:42.161124 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:42.161106 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:43.160029 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:43.159994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:43.160530 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:43.159999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:43.160530 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:43.160149 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:43.160530 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:43.160201 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:44.160969 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:44.160940 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:44.161383 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:44.161047 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:45.160947 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:45.160912 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:45.161122 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:45.160957 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:45.161122 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:45.161055 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:45.161560 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:45.161179 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:46.160394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:46.160363 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:46.160541 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:46.160476 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:47.160033 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:47.159998 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:47.160507 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:47.160002 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:47.160507 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:47.160132 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:47.160507 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:47.160209 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:48.160154 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:48.160117 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:48.160669 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.160237 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:48.799715 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:48.799671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:48.799900 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:48.799817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:48.799900 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.799854 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:48.800013 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.799941 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:04.799918725 +0000 UTC m=+34.141502334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:54:48.800013 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.799968 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:54:48.800013 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.799987 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:54:48.800013 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.800001 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:48.800162 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.800055 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:04.800039344 +0000 UTC m=+34.141622977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:54:48.900277 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:48.900230 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:48.900462 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.900377 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:48.900462 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:48.900460 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:55:04.900438649 +0000 UTC m=+34.242022288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:54:49.160689 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:49.160606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:49.161107 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:49.160606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:49.161107 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:49.160723 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:49.161107 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:49.160846 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:50.160248 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:50.160213 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:50.160434 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:50.160343 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:51.161425 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:51.161386 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:51.161927 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:51.161499 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:51.161927 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:51.161533 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:51.161927 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:51.161577 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:52.160661 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.160449 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:52.160848 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:52.160822 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:52.383363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.383339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" event={"ID":"2de2a3e1acdc244afa40a51cd31b1009","Type":"ContainerStarted","Data":"95530d755f5170049d566ff579aaa56ecae2d60562aee14698a96309674a5173"} Jul 31 19:54:52.384992 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.384974 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:54:52.385298 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.385281 2556 generic.go:356] "Generic (PLEG): container finished" podID="806b7291-7bec-44fd-942d-8e0ec25d7ab8" containerID="c70466623346d03c8e57d51524b4fb7fd8bbffa6985feddc1af84c01224ec4c7" exitCode=1 Jul 31 19:54:52.385365 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.385349 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"f996e84bfc1a5e6557e8d4a38f5ed4b73a4052fcce23a062b7cd5db7dbcfae50"} Jul 31 19:54:52.385437 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.385420 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerDied","Data":"c70466623346d03c8e57d51524b4fb7fd8bbffa6985feddc1af84c01224ec4c7"} Jul 31 19:54:52.385494 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.385476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"83ed06139a7fddcff4ff3f332b59380510de259d475980f81aacab4f63ece450"} Jul 31 19:54:52.386755 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.386739 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c2l2b" event={"ID":"09eabf70-c174-4ddd-958c-8b43daed39dd","Type":"ContainerStarted","Data":"150603771f05b859732c9187c32857e260488f1aa82bf5b163213929d3f7e413"} Jul 31 19:54:52.388084 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.388063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerStarted","Data":"37359c3898f6955986aa42252c595cc26d05645e6a5af0c2fde4311c5e7a0afc"} Jul 31 19:54:52.389363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.389344 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5d27" event={"ID":"0af497c1-b530-46e6-861d-8324d9d3d829","Type":"ContainerStarted","Data":"98bc845e6b9f3f170b54a0f68b48739ceb7e17397287e6b563d81ab9717da45e"} Jul 31 19:54:52.390509 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.390490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9t5t" event={"ID":"4b788b4e-4803-4515-bba2-39b870e004f7","Type":"ContainerStarted","Data":"2d9b96fff0450a35c51825d83baaa5ece933bbd1eb86d2f2c0473dff86a6c790"} Jul 31 19:54:52.391830 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.391810 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" event={"ID":"716ba6f1-ffa5-4f25-94d6-d4931699f301","Type":"ContainerStarted","Data":"c77ed5b9e889c0d49ac89e6ff8d603dd7e1441b1d4eddf7f6ebf7f5a8a7cbbd1"} Jul 31 19:54:52.393103 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.393084 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" event={"ID":"1b7466ef-1f28-432c-bc6d-93b443cc01db","Type":"ContainerStarted","Data":"0b799dae9eee1d0dfd90c42795fca63187de2e99500e888d94aa8a2241395123"} Jul 31 19:54:52.398831 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.398789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-178.ec2.internal" podStartSLOduration=20.398778681 podStartE2EDuration="20.398778681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:35.365254304 +0000 UTC m=+4.706837931" watchObservedRunningTime="2026-07-31 19:54:52.398778681 +0000 UTC m=+21.740362306" Jul 31 19:54:52.399040 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.399020 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-178.ec2.internal" podStartSLOduration=20.399016433 podStartE2EDuration="20.399016433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:52.398176928 +0000 UTC m=+21.739760554" watchObservedRunningTime="2026-07-31 19:54:52.399016433 +0000 UTC m=+21.740600058" Jul 31 19:54:52.412756 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.412723 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c9t5t" podStartSLOduration=2.865993466 podStartE2EDuration="21.412711731s" totalImagesPullingTime="18.546718265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:52.412597402 +0000 UTC m=+21.754181038" watchObservedRunningTime="2026-07-31 19:54:52.412711731 +0000 UTC m=+21.754295400" Jul 31 19:54:52.425080 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.425041 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h5d27" podStartSLOduration=2.889070084 podStartE2EDuration="21.425030632s" totalImagesPullingTime="18.535960548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:52.424622684 +0000 UTC m=+21.766206310" watchObservedRunningTime="2026-07-31 19:54:52.425030632 +0000 UTC m=+21.766614257" Jul 31 19:54:52.458682 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.458639 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sfl2b" podStartSLOduration=2.896727864 podStartE2EDuration="21.458622579s" totalImagesPullingTime="18.561894715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:52.458029364 +0000 UTC m=+21.799612990" watchObservedRunningTime="2026-07-31 19:54:52.458622579 +0000 UTC m=+21.800206205" Jul 31 19:54:52.477073 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:52.474237 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c2l2b" podStartSLOduration=2.915415779 podStartE2EDuration="21.474221803s" totalImagesPullingTime="18.558806024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:52.473417279 +0000 UTC m=+21.815000905" watchObservedRunningTime="2026-07-31 19:54:52.474221803 +0000 UTC m=+21.815805429" Jul 31 19:54:53.160546 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.160384 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:53.160709 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.160385 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:53.160753 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:53.160727 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:53.160795 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:53.160616 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:53.398977 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.398955 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:54:53.399811 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.399699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"fe6afc58f6c323ba3f480e74ec7a743a321f25415beb0726e4abcb5dc6f45b23"} Jul 31 19:54:53.399811 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.399782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"df59382d764698ff11a9434b9e3668646bea4a60f32a62d6b0e5919322ca7900"} Jul 31 19:54:53.399923 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.399820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"74c734b3559c6a4c2760b57f7543d492029a92d6b505fef8a99eb80700315023"} Jul 31 19:54:53.401241 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.401219 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="37359c3898f6955986aa42252c595cc26d05645e6a5af0c2fde4311c5e7a0afc" exitCode=0 Jul 31 19:54:53.401353 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.401298 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"37359c3898f6955986aa42252c595cc26d05645e6a5af0c2fde4311c5e7a0afc"} Jul 31 19:54:53.403245 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.403220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fn5g2" event={"ID":"52ae78ac-54e5-49b8-8447-8f7f17fd318a","Type":"ContainerStarted","Data":"eac8d0743532215d2556fdab738a43e9c2c3f7513527b2c13110b01b614111e6"} Jul 31 19:54:53.404811 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.404790 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcg55" event={"ID":"4f5f107e-99d4-4459-82b5-49dcdb626678","Type":"ContainerStarted","Data":"844d3a9c3434989040dcf6ce7b5dbd71f81baae1436b748b3662a42959d89c73"} Jul 31 19:54:53.437279 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.437241 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fn5g2" podStartSLOduration=3.39825795 podStartE2EDuration="22.437229663s" totalImagesPullingTime="19.038971713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:53.436532736 +0000 UTC m=+22.778116363" watchObservedRunningTime="2026-07-31 19:54:53.437229663 +0000 UTC m=+22.778813289" Jul 31 19:54:53.449387 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.449355 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hcg55" podStartSLOduration=3.8911688460000002 podStartE2EDuration="22.449343015s" totalImagesPullingTime="18.558174169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:53.448974866 +0000 UTC m=+22.790558491" watchObservedRunningTime="2026-07-31 19:54:53.449343015 +0000 UTC m=+22.790926641" Jul 31 19:54:53.484368 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:53.484349 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 19:54:54.122921 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:54.122802 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T19:54:53.484364097Z","UUID":"f575ecdb-a7b7-4336-8b87-e2a0c09e3bd4","Handler":null,"Name":"","Endpoint":""} Jul 31 19:54:54.124513 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:54.124491 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 19:54:54.124633 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:54.124532 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 19:54:54.160486 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:54.160455 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:54.160647 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:54.160585 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:54.409709 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:54.409636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" event={"ID":"1b7466ef-1f28-432c-bc6d-93b443cc01db","Type":"ContainerStarted","Data":"a82714e1926dea8c39a43e620accf6fe365a67a09ec662a25c78a83d29a4a3b3"} Jul 31 19:54:55.161053 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.160836 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:55.161191 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.160904 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:55.161191 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:55.161094 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:55.161191 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:55.161163 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:55.414820 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.414732 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" event={"ID":"1b7466ef-1f28-432c-bc6d-93b443cc01db","Type":"ContainerStarted","Data":"90a2dfbb8ebc77855f5bd7a0787e40488b751a5b0883cf0a614776da14e3906f"} Jul 31 19:54:55.418231 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.418212 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:54:55.418655 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.418625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"ae6687ac9475449d03867d0b4b3927c26bcd98519b7ac9f983188b29b07b6833"} Jul 31 19:54:55.431667 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:55.431620 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nfz5" podStartSLOduration=3.647832439 podStartE2EDuration="24.431602964s" totalImagesPullingTime="20.783770525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:55.430657137 +0000 UTC m=+24.772240765" watchObservedRunningTime="2026-07-31 19:54:55.431602964 +0000 UTC m=+24.773186592" Jul 31 19:54:56.160900 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:56.160872 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:56.161076 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:56.160971 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:57.160810 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.160734 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:57.161484 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:57.160874 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:57.161484 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.160917 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:57.161484 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:57.160994 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:57.206160 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.206126 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:57.206724 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.206695 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:57.424026 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.423953 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:57.424481 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:57.424451 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c9t5t" Jul 31 19:54:58.160920 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.160767 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:54:58.161360 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:58.160993 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:54:58.429199 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.429132 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:54:58.429533 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.429511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"637d83b61db40302e5cc9773c635c9447732595dd6e2ea8028a5cf4e43260f47"} Jul 31 19:54:58.429780 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.429757 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:58.429911 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.429848 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:58.429963 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.429945 2556 scope.go:117] "RemoveContainer" containerID="c70466623346d03c8e57d51524b4fb7fd8bbffa6985feddc1af84c01224ec4c7" Jul 31 19:54:58.431409 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.431385 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="dc4b8d48b77f6036efab9e9788d86c96bb728206c946d663c5f191334505689e" exitCode=0 Jul 31 19:54:58.431491 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.431449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"dc4b8d48b77f6036efab9e9788d86c96bb728206c946d663c5f191334505689e"} Jul 31 19:54:58.445323 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:58.445289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:59.160921 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.160883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:54:59.160921 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.160903 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:54:59.161567 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:59.161026 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:54:59.161567 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:54:59.161147 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:54:59.438939 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.438901 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:54:59.440296 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.439403 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" event={"ID":"806b7291-7bec-44fd-942d-8e0ec25d7ab8","Type":"ContainerStarted","Data":"c2b0969f6ba5f63b9442806084737aca423a56673a55586a2765511804996d07"} Jul 31 19:54:59.440296 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.439831 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:59.460174 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.460143 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:54:59.469107 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:54:59.469051 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" podStartSLOduration=9.87451958 podStartE2EDuration="28.469031649s" totalImagesPullingTime="18.594512069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:54:59.467558736 +0000 UTC m=+28.809142361" watchObservedRunningTime="2026-07-31 19:54:59.469031649 +0000 UTC m=+28.810615275" Jul 31 19:55:00.058520 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.058491 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hf82d"] Jul 31 19:55:00.058681 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.058590 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:00.058735 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:00.058680 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:55:00.061463 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.061436 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tncq5"] Jul 31 19:55:00.061584 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.061543 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:00.061662 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:00.061641 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:55:00.062042 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.061999 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qn4bd"] Jul 31 19:55:00.062181 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.062088 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:00.062181 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:00.062170 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:55:00.443752 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.443716 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="532f5cd0d94b7a2887125c2e2a0763fee39e126d6ccb36a4ca79df41d90b1bd6" exitCode=0 Jul 31 19:55:00.444102 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:00.443798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"532f5cd0d94b7a2887125c2e2a0763fee39e126d6ccb36a4ca79df41d90b1bd6"} Jul 31 19:55:02.160299 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:02.160268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:02.160299 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:02.160291 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:02.160908 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:02.160268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:02.160908 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:02.160423 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:55:02.160908 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:02.160472 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:55:02.160908 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:02.160548 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:55:02.450721 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:02.450640 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="9e765f9bb75738cff6546e80bf2cf7d1dec7ccd801f162abf4150d14e43ad354" exitCode=0 Jul 31 19:55:02.450721 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:02.450699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"9e765f9bb75738cff6546e80bf2cf7d1dec7ccd801f162abf4150d14e43ad354"} Jul 31 19:55:04.160858 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.160825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:04.161390 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.160824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:04.161390 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.160944 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qn4bd" podUID="7a481939-8e1f-4acd-9361-e7095e8d52f3" Jul 31 19:55:04.161390 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.160824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:04.161390 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.161001 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tncq5" podUID="84ef4519-773b-47ba-b51f-aaf05966fd13" Jul 31 19:55:04.161390 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.161086 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf82d" podUID="e9584fe9-248f-4aa7-8dd0-2633a9bf857f" Jul 31 19:55:04.819905 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.819876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:04.820101 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.819951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:04.820101 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820055 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:55:04.820101 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820061 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 19:55:04.820101 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820082 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 19:55:04.820101 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820093 2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gq57 for pod openshift-network-diagnostics/network-check-target-tncq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:55:04.820354 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820119 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:36.820099251 +0000 UTC m=+66.161682858 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 19:55:04.820354 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.820138 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57 podName:84ef4519-773b-47ba-b51f-aaf05966fd13 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:36.820125523 +0000 UTC m=+66.161709127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gq57" (UniqueName: "kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57") pod "network-check-target-tncq5" (UID: "84ef4519-773b-47ba-b51f-aaf05966fd13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 19:55:04.920427 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:04.920394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:04.920585 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.920529 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 19:55:04.920643 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:04.920594 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret podName:e9584fe9-248f-4aa7-8dd0-2633a9bf857f nodeName:}" failed. No retries permitted until 2026-07-31 19:55:36.920574379 +0000 UTC m=+66.262157995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret") pod "global-pull-secret-syncer-hf82d" (UID: "e9584fe9-248f-4aa7-8dd0-2633a9bf857f") : object "kube-system"/"original-pull-secret" not registered Jul 31 19:55:05.010014 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.009986 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-178.ec2.internal" event="NodeReady" Jul 31 19:55:05.010178 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.010133 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 19:55:05.044710 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.044664 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:55:05.049334 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.049293 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z"] Jul 31 19:55:05.049463 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.049409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.051585 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.051563 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 31 19:55:05.051701 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.051630 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 31 19:55:05.051701 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.051636 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 31 19:55:05.051845 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.051726 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dg6cw\"" Jul 31 19:55:05.053373 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.053352 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5"] Jul 31 19:55:05.053476 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.053457 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.055947 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.055920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 31 19:55:05.055947 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.055936 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 19:55:05.056108 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.055952 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 19:55:05.056548 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.056528 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 19:55:05.057262 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.057239 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2"] Jul 31 19:55:05.057363 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.057294 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 31 19:55:05.057422 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.057363 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.059810 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.059791 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ttrb5\"" Jul 31 19:55:05.060290 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.060275 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 31 19:55:05.064803 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.064763 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.065695 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.065671 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z"] Jul 31 19:55:05.066606 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.066570 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5"] Jul 31 19:55:05.067166 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067021 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 31 19:55:05.067295 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067280 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 31 19:55:05.067384 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067362 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:55:05.067442 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067406 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 31 19:55:05.067493 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067404 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 31 19:55:05.067908 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.067891 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-48wns"] Jul 31 19:55:05.073351 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.073286 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2"] Jul 31 19:55:05.073351 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.073325 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hgjn8"] Jul 31 19:55:05.073495 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.073409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.075546 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.075531 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 19:55:05.075624 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.075551 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 19:55:05.075624 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.075557 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6742\"" Jul 31 19:55:05.076800 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.076783 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.079170 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.079152 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48wns"] Jul 31 19:55:05.079249 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.079211 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 19:55:05.079249 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.079243 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 19:55:05.079461 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.079269 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vs8lm\"" Jul 31 19:55:05.079503 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.079490 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 19:55:05.083791 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.083774 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgjn8"] Jul 31 19:55:05.222847 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.222810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.222894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.222922 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.222965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39865e8b-0850-42f1-8f5e-568bbd60125e-klusterlet-config\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.222989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae9309-4896-448f-b68d-0bbcb57bd552-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223034 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39865e8b-0850-42f1-8f5e-568bbd60125e-tmp\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzz5\" (UniqueName: \"kubernetes.io/projected/4d6a17d3-d7ff-46c5-96ce-011636586158-kube-api-access-mnzz5\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223090 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223117 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57414e35-918a-41db-a3ad-cd04428a36dd-config-volume\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223146 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57414e35-918a-41db-a3ad-cd04428a36dd-tmp-dir\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.223243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d6a17d3-d7ff-46c5-96ce-011636586158-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223418 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgb7p\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjs97\" (UniqueName: \"kubernetes.io/projected/57414e35-918a-41db-a3ad-cd04428a36dd-kube-api-access-sjs97\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223502 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp597\" (UniqueName: \"kubernetes.io/projected/c2ae9309-4896-448f-b68d-0bbcb57bd552-kube-api-access-gp597\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj95\" (UniqueName: \"kubernetes.io/projected/04080c1f-cb16-4c9c-96f3-baf406112fc2-kube-api-access-pzj95\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223570 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqr9\" (UniqueName: \"kubernetes.io/projected/39865e8b-0850-42f1-8f5e-568bbd60125e-kube-api-access-qfqr9\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223616 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.223777 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.223673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04080c1f-cb16-4c9c-96f3-baf406112fc2-cert\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.324824 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.324735 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.324824 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.324775 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.324824 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.324810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39865e8b-0850-42f1-8f5e-568bbd60125e-klusterlet-config\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.325090 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.324946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae9309-4896-448f-b68d-0bbcb57bd552-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.325090 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.324994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39865e8b-0850-42f1-8f5e-568bbd60125e-tmp\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.325090 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzz5\" (UniqueName: \"kubernetes.io/projected/4d6a17d3-d7ff-46c5-96ce-011636586158-kube-api-access-mnzz5\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.325090 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.325090 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57414e35-918a-41db-a3ad-cd04428a36dd-config-volume\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57414e35-918a-41db-a3ad-cd04428a36dd-tmp-dir\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325151 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325196 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d6a17d3-d7ff-46c5-96ce-011636586158-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325273 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgb7p\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjs97\" (UniqueName: \"kubernetes.io/projected/57414e35-918a-41db-a3ad-cd04428a36dd-kube-api-access-sjs97\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.325360 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325345 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp597\" (UniqueName: \"kubernetes.io/projected/c2ae9309-4896-448f-b68d-0bbcb57bd552-kube-api-access-gp597\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325374 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj95\" (UniqueName: \"kubernetes.io/projected/04080c1f-cb16-4c9c-96f3-baf406112fc2-kube-api-access-pzj95\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325394 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39865e8b-0850-42f1-8f5e-568bbd60125e-tmp\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqr9\" (UniqueName: \"kubernetes.io/projected/39865e8b-0850-42f1-8f5e-568bbd60125e-kube-api-access-qfqr9\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325444 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325524 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04080c1f-cb16-4c9c-96f3-baf406112fc2-cert\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325553 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325581 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325663 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.325738 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.325751 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.325810 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:05.825792642 +0000 UTC m=+35.167376253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:05.325880 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.325844 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57414e35-918a-41db-a3ad-cd04428a36dd-tmp-dir\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.326559 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.326520 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ae9309-4896-448f-b68d-0bbcb57bd552-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.328540 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.328262 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57414e35-918a-41db-a3ad-cd04428a36dd-config-volume\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.330637 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.330577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.330756 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.330726 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d6a17d3-d7ff-46c5-96ce-011636586158-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.330960 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.330931 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.331046 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.331031 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:05.331105 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.331039 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.331156 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.331110 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:05.831090828 +0000 UTC m=+35.172674437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:05.331774 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.331746 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.331774 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.331767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.332196 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.332173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04080c1f-cb16-4c9c-96f3-baf406112fc2-cert\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.332281 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.332253 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.332712 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.332684 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39865e8b-0850-42f1-8f5e-568bbd60125e-klusterlet-config\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.332858 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.332840 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2ae9309-4896-448f-b68d-0bbcb57bd552-ca\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.337048 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.337018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.337128 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.337029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzz5\" (UniqueName: \"kubernetes.io/projected/4d6a17d3-d7ff-46c5-96ce-011636586158-kube-api-access-mnzz5\") pod \"managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5\" (UID: \"4d6a17d3-d7ff-46c5-96ce-011636586158\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.339205 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.339185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp597\" (UniqueName: \"kubernetes.io/projected/c2ae9309-4896-448f-b68d-0bbcb57bd552-kube-api-access-gp597\") pod \"cluster-proxy-proxy-agent-78bb7597cf-425z2\" (UID: \"c2ae9309-4896-448f-b68d-0bbcb57bd552\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.339322 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.339293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqr9\" (UniqueName: \"kubernetes.io/projected/39865e8b-0850-42f1-8f5e-568bbd60125e-kube-api-access-qfqr9\") pod \"klusterlet-addon-workmgr-864f9b9db5-qcp7z\" (UID: \"39865e8b-0850-42f1-8f5e-568bbd60125e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.339610 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.339593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj95\" (UniqueName: \"kubernetes.io/projected/04080c1f-cb16-4c9c-96f3-baf406112fc2-kube-api-access-pzj95\") pod \"ingress-canary-hgjn8\" (UID: \"04080c1f-cb16-4c9c-96f3-baf406112fc2\") " pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.342554 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.342530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjs97\" (UniqueName: \"kubernetes.io/projected/57414e35-918a-41db-a3ad-cd04428a36dd-kube-api-access-sjs97\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.342927 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.342912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgb7p\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.373002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.372863 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:05.386490 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.386456 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" Jul 31 19:55:05.393669 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.393649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:55:05.416732 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.416704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgjn8" Jul 31 19:55:05.830530 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.830485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:05.830696 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.830642 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:05.830696 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.830661 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:05.830806 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.830729 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:06.83070637 +0000 UTC m=+36.172289991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:05.931449 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:05.931407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:05.931641 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.931592 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:05.931703 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:05.931680 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:06.931660022 +0000 UTC m=+36.273243632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:06.160747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.160670 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:06.160747 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.160685 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:06.160941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.160856 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:06.163585 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.163561 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 19:55:06.163701 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.163614 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 19:55:06.163822 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.163803 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 19:55:06.164384 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.164369 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9snhb\"" Jul 31 19:55:06.164600 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.164587 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twv2l\"" Jul 31 19:55:06.164729 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.164698 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 19:55:06.839691 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.839656 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:06.840075 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:06.839795 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:06.840075 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:06.839812 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:06.840075 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:06.839875 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:08.839855758 +0000 UTC m=+38.181439364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:06.940971 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.940924 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:06.941139 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:06.941094 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:06.941195 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:06.941157 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:08.941139387 +0000 UTC m=+38.282723015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:06.965079 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.964969 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2"] Jul 31 19:55:06.969931 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.969910 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z"] Jul 31 19:55:06.973345 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.973303 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgjn8"] Jul 31 19:55:06.974057 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:06.974031 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ae9309_4896_448f_b68d_0bbcb57bd552.slice/crio-f5f7d780e6c61967c572867e6b913d380ff9b6c82eb7ba7b1e84174e0ab58078 WatchSource:0}: Error finding container f5f7d780e6c61967c572867e6b913d380ff9b6c82eb7ba7b1e84174e0ab58078: Status 404 returned error can't find the container with id f5f7d780e6c61967c572867e6b913d380ff9b6c82eb7ba7b1e84174e0ab58078 Jul 31 19:55:06.974393 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:06.974324 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39865e8b_0850_42f1_8f5e_568bbd60125e.slice/crio-4fe3b29c645071ba7cfc6aed93e2a088ac34c144015abd9bfdd9d603dc03b848 WatchSource:0}: Error finding container 4fe3b29c645071ba7cfc6aed93e2a088ac34c144015abd9bfdd9d603dc03b848: Status 404 returned error can't find the container with id 4fe3b29c645071ba7cfc6aed93e2a088ac34c144015abd9bfdd9d603dc03b848 Jul 31 19:55:06.976120 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:06.975973 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04080c1f_cb16_4c9c_96f3_baf406112fc2.slice/crio-ebf6c270bebd2717cef8ce99c5fe7e4f1a55bdb8a59b6657234e8e7e85625480 WatchSource:0}: Error finding container ebf6c270bebd2717cef8ce99c5fe7e4f1a55bdb8a59b6657234e8e7e85625480: Status 404 returned error can't find the container with id ebf6c270bebd2717cef8ce99c5fe7e4f1a55bdb8a59b6657234e8e7e85625480 Jul 31 19:55:06.982105 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:06.982081 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5"] Jul 31 19:55:06.985118 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:06.985084 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d6a17d3_d7ff_46c5_96ce_011636586158.slice/crio-cbd1e422e7fd3c880712cb7bd7a12ef72cc2b4cc3c5ac59582828b21c9665d10 WatchSource:0}: Error finding container cbd1e422e7fd3c880712cb7bd7a12ef72cc2b4cc3c5ac59582828b21c9665d10: Status 404 returned error can't find the container with id cbd1e422e7fd3c880712cb7bd7a12ef72cc2b4cc3c5ac59582828b21c9665d10 Jul 31 19:55:07.463108 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.463025 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" event={"ID":"4d6a17d3-d7ff-46c5-96ce-011636586158","Type":"ContainerStarted","Data":"cbd1e422e7fd3c880712cb7bd7a12ef72cc2b4cc3c5ac59582828b21c9665d10"} Jul 31 19:55:07.463969 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.463942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerStarted","Data":"f5f7d780e6c61967c572867e6b913d380ff9b6c82eb7ba7b1e84174e0ab58078"} Jul 31 19:55:07.464831 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.464815 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" event={"ID":"39865e8b-0850-42f1-8f5e-568bbd60125e","Type":"ContainerStarted","Data":"4fe3b29c645071ba7cfc6aed93e2a088ac34c144015abd9bfdd9d603dc03b848"} Jul 31 19:55:07.467174 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.467151 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="4b1f15498b64b15a23bc3b66b0a476bf6bfbd2de39fa0f85e65f0ef2dd02d458" exitCode=0 Jul 31 19:55:07.467243 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.467219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"4b1f15498b64b15a23bc3b66b0a476bf6bfbd2de39fa0f85e65f0ef2dd02d458"} Jul 31 19:55:07.468301 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:07.468278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgjn8" event={"ID":"04080c1f-cb16-4c9c-96f3-baf406112fc2","Type":"ContainerStarted","Data":"ebf6c270bebd2717cef8ce99c5fe7e4f1a55bdb8a59b6657234e8e7e85625480"} Jul 31 19:55:08.489752 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:08.489432 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4e72d7e-54cd-404c-817c-eb5baa46fe50" containerID="0dff9fd4d96c0cc3af82bf7339f79928bdd5ef9c5ee78af543d476c014060a0f" exitCode=0 Jul 31 19:55:08.489752 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:08.489683 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerDied","Data":"0dff9fd4d96c0cc3af82bf7339f79928bdd5ef9c5ee78af543d476c014060a0f"} Jul 31 19:55:08.862091 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:08.862041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:08.862286 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:08.862268 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:08.862379 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:08.862289 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:08.862457 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:08.862393 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:12.862373638 +0000 UTC m=+42.203957249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:08.963265 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:08.963227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:08.963461 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:08.963443 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:08.963535 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:08.963515 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:12.963495739 +0000 UTC m=+42.305079351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:09.500607 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:09.500568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" event={"ID":"e4e72d7e-54cd-404c-817c-eb5baa46fe50","Type":"ContainerStarted","Data":"bace60f5b063df878d4d4e72faac2ab50550f2d58a6471e4425edc613830e56d"} Jul 31 19:55:12.891127 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:12.891081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:12.891573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:12.891256 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:12.891573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:12.891274 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:12.891573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:12.891350 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:20.891334812 +0000 UTC m=+50.232918440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:12.991545 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:12.991506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:12.991722 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:12.991696 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:12.991791 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:12.991777 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:20.991756564 +0000 UTC m=+50.333340172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:16.519865 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.519829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgjn8" event={"ID":"04080c1f-cb16-4c9c-96f3-baf406112fc2","Type":"ContainerStarted","Data":"2a62437cc85d4c95db851f3d94cde41588ec353c70b41817f633e9982801582d"} Jul 31 19:55:16.521421 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.521397 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" event={"ID":"4d6a17d3-d7ff-46c5-96ce-011636586158","Type":"ContainerStarted","Data":"d87797afb4f06635c91b8de0beb55a39d2a6dc104b3f55ba93c1bc6b42e0a386"} Jul 31 19:55:16.526565 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.526534 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerStarted","Data":"6c1d569f70cc14343bfc11ee47beb58d0c5e92ce4c0b2da9afa023f200290c78"} Jul 31 19:55:16.526687 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.526574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerStarted","Data":"4bffd868d44b48e68ab9e81acee192ec9634855d6ac6672a1671d4ef0cac2e41"} Jul 31 19:55:16.526687 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.526589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerStarted","Data":"da32f774a5bcc9967833ba14fa4eaa43b05af757472f912c62eb85bb9c56ef31"} Jul 31 19:55:16.527763 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.527742 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" event={"ID":"39865e8b-0850-42f1-8f5e-568bbd60125e","Type":"ContainerStarted","Data":"ce827f043f95b710654b7dbd12ed0d610e58959c38157d3fb8304b6c4dddd213"} Jul 31 19:55:16.527954 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.527937 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:16.529614 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.529596 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" Jul 31 19:55:16.536103 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.536066 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k6h2z" podStartSLOduration=15.721231504 podStartE2EDuration="45.536051779s" totalImagesPullingTime="29.814820275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:09.526032671 +0000 UTC m=+38.867616295" watchObservedRunningTime="2026-07-31 19:55:16.536051779 +0000 UTC m=+45.877635407" Jul 31 19:55:16.536711 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.536679 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hgjn8" podStartSLOduration=2.733102422 podStartE2EDuration="11.536670802s" totalImagesPullingTime="8.80356838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:16.535913661 +0000 UTC m=+45.877497287" watchObservedRunningTime="2026-07-31 19:55:16.536670802 +0000 UTC m=+45.878254418" Jul 31 19:55:16.550526 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.550486 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864f9b9db5-qcp7z" podStartSLOduration=19.744780553 podStartE2EDuration="28.550472393s" totalImagesPullingTime="8.80569184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:16.550057633 +0000 UTC m=+45.891641260" watchObservedRunningTime="2026-07-31 19:55:16.550472393 +0000 UTC m=+45.892056019" Jul 31 19:55:16.565612 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.565566 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6597cb9fdd-tjjs5" podStartSLOduration=19.76363919 podStartE2EDuration="28.565555695s" totalImagesPullingTime="8.801916505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:16.564715986 +0000 UTC m=+45.906299625" watchObservedRunningTime="2026-07-31 19:55:16.565555695 +0000 UTC m=+45.907139320" Jul 31 19:55:16.581677 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:16.581634 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" podStartSLOduration=19.776290621 podStartE2EDuration="28.581617845s" totalImagesPullingTime="8.805327224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:16.581194885 +0000 UTC m=+45.922778513" watchObservedRunningTime="2026-07-31 19:55:16.581617845 +0000 UTC m=+45.923201473" Jul 31 19:55:20.945217 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:20.945173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:20.945705 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:20.945394 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:20.945705 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:20.945423 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:20.945705 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:20.945530 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:36.945507042 +0000 UTC m=+66.287090652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:21.046517 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:21.046482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:21.046665 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:21.046616 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:21.046708 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:21.046680 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:55:37.046666667 +0000 UTC m=+66.388250275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:31.466741 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:31.466713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wncgg" Jul 31 19:55:36.860519 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.860466 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:55:36.861066 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.860540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:36.862915 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.862890 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 19:55:36.862915 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.862912 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 19:55:36.870881 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:36.870864 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 19:55:36.870928 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:36.870913 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs podName:7a481939-8e1f-4acd-9361-e7095e8d52f3 nodeName:}" failed. No retries permitted until 2026-07-31 19:56:40.870899282 +0000 UTC m=+130.212482887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs") pod "network-metrics-daemon-qn4bd" (UID: "7a481939-8e1f-4acd-9361-e7095e8d52f3") : secret "metrics-daemon-secret" not found Jul 31 19:55:36.872819 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.872806 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 19:55:36.885166 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.885147 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq57\" (UniqueName: \"kubernetes.io/projected/84ef4519-773b-47ba-b51f-aaf05966fd13-kube-api-access-7gq57\") pod \"network-check-target-tncq5\" (UID: \"84ef4519-773b-47ba-b51f-aaf05966fd13\") " pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:36.960952 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.960930 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:55:36.961042 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.960984 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:36.961088 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:36.961064 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 19:55:36.961088 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:36.961079 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6457b674-ntd74: secret "image-registry-tls" not found Jul 31 19:55:36.961163 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:36.961121 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls podName:390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96 nodeName:}" failed. No retries permitted until 2026-07-31 19:56:08.961108673 +0000 UTC m=+98.302692277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls") pod "image-registry-6d6457b674-ntd74" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96") : secret "image-registry-tls" not found Jul 31 19:55:36.963531 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.963517 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 19:55:36.973432 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:36.973412 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9584fe9-248f-4aa7-8dd0-2633a9bf857f-original-pull-secret\") pod \"global-pull-secret-syncer-hf82d\" (UID: \"e9584fe9-248f-4aa7-8dd0-2633a9bf857f\") " pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:37.062124 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.062046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:55:37.062287 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.062173 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 19:55:37.062287 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.062237 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls podName:57414e35-918a-41db-a3ad-cd04428a36dd nodeName:}" failed. No retries permitted until 2026-07-31 19:56:09.062219702 +0000 UTC m=+98.403803308 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls") pod "dns-default-48wns" (UID: "57414e35-918a-41db-a3ad-cd04428a36dd") : secret "dns-default-metrics-tls" not found Jul 31 19:55:37.080616 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.080596 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9snhb\"" Jul 31 19:55:37.082514 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.082492 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gqrm2"] Jul 31 19:55:37.083445 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.083420 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf82d" Jul 31 19:55:37.085060 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.085043 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.087347 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.087323 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 19:55:37.087460 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.087369 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j929d\"" Jul 31 19:55:37.087460 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.087378 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 19:55:37.087549 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.087507 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 19:55:37.087593 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.087576 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 19:55:37.088785 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.088767 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:37.095701 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.095670 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqrm2"] Jul 31 19:55:37.162408 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.162370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.162571 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.162411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.162571 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.162452 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-api-access-6rjcc\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.162571 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.162473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81566aae-bac4-44f8-b062-5e88eeaa90a3-crio-socket\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.162571 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.162545 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81566aae-bac4-44f8-b062-5e88eeaa90a3-data-volume\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.212649 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.212609 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hf82d"] Jul 31 19:55:37.215658 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:37.215631 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9584fe9_248f_4aa7_8dd0_2633a9bf857f.slice/crio-3eb8867a2fad294188d4b43c992e984e3bb3e89bfbc636ade00875a0093863b0 WatchSource:0}: Error finding container 3eb8867a2fad294188d4b43c992e984e3bb3e89bfbc636ade00875a0093863b0: Status 404 returned error can't find the container with id 3eb8867a2fad294188d4b43c992e984e3bb3e89bfbc636ade00875a0093863b0 Jul 31 19:55:37.227729 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.227696 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tncq5"] Jul 31 19:55:37.229810 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:37.229787 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ef4519_773b_47ba_b51f_aaf05966fd13.slice/crio-a40ef6616891fd883332740f32a293d0efbd12e3c2e443845807676f3163f792 WatchSource:0}: Error finding container a40ef6616891fd883332740f32a293d0efbd12e3c2e443845807676f3163f792: Status 404 returned error can't find the container with id a40ef6616891fd883332740f32a293d0efbd12e3c2e443845807676f3163f792 Jul 31 19:55:37.263884 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.263859 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.263963 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.263887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.263963 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.263920 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-api-access-6rjcc\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.264031 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.263987 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 19:55:37.264066 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.264043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81566aae-bac4-44f8-b062-5e88eeaa90a3-crio-socket\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.264066 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.264059 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:37.764038506 +0000 UTC m=+67.105622124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : configmap "kube-rbac-proxy" not found Jul 31 19:55:37.264131 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.264100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81566aae-bac4-44f8-b062-5e88eeaa90a3-crio-socket\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.264131 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.264101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81566aae-bac4-44f8-b062-5e88eeaa90a3-data-volume\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.264270 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.264254 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 19:55:37.264306 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.264298 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:37.764285847 +0000 UTC m=+67.105869452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : secret "insights-runtime-extractor-tls" not found Jul 31 19:55:37.264378 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.264364 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81566aae-bac4-44f8-b062-5e88eeaa90a3-data-volume\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.272144 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.272128 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjcc\" (UniqueName: \"kubernetes.io/projected/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-api-access-6rjcc\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.592025 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.591993 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hf82d" event={"ID":"e9584fe9-248f-4aa7-8dd0-2633a9bf857f","Type":"ContainerStarted","Data":"3eb8867a2fad294188d4b43c992e984e3bb3e89bfbc636ade00875a0093863b0"} Jul 31 19:55:37.593024 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.592997 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tncq5" event={"ID":"84ef4519-773b-47ba-b51f-aaf05966fd13","Type":"ContainerStarted","Data":"a40ef6616891fd883332740f32a293d0efbd12e3c2e443845807676f3163f792"} Jul 31 19:55:37.768603 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.768573 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.768750 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:37.768610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:37.768750 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.768691 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 19:55:37.768819 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.768751 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:38.768737972 +0000 UTC m=+68.110321577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : configmap "kube-rbac-proxy" not found Jul 31 19:55:37.768819 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.768759 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 19:55:37.768819 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:37.768802 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:38.768787067 +0000 UTC m=+68.110370671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : secret "insights-runtime-extractor-tls" not found Jul 31 19:55:38.778691 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:38.778525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:38.778691 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:38.778580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:38.778691 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:38.778655 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 19:55:38.779271 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:38.778723 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:40.778704974 +0000 UTC m=+70.120288599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : configmap "kube-rbac-proxy" not found Jul 31 19:55:38.779271 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:38.778760 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 19:55:38.779271 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:38.778824 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:40.778807164 +0000 UTC m=+70.120390773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : secret "insights-runtime-extractor-tls" not found Jul 31 19:55:39.057492 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:39.057430 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c2l2b_09eabf70-c174-4ddd-958c-8b43daed39dd/dns-node-resolver/0.log" Jul 31 19:55:39.856987 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:39.856953 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5d27_0af497c1-b530-46e6-861d-8324d9d3d829/node-ca/0.log" Jul 31 19:55:40.794846 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:40.794810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:40.794846 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:40.794855 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:40.795103 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:40.794953 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 19:55:40.795103 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:40.794975 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 19:55:40.795103 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:40.795030 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:44.795009009 +0000 UTC m=+74.136592615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : configmap "kube-rbac-proxy" not found Jul 31 19:55:40.795103 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:40.795052 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:44.795041891 +0000 UTC m=+74.136625495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : secret "insights-runtime-extractor-tls" not found Jul 31 19:55:40.857858 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:40.857826 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgjn8_04080c1f-cb16-4c9c-96f3-baf406112fc2/serve-healthcheck-canary/0.log" Jul 31 19:55:42.611759 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:42.611724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hf82d" event={"ID":"e9584fe9-248f-4aa7-8dd0-2633a9bf857f","Type":"ContainerStarted","Data":"620b79c312b84d4c1da3ba9606a666ca27e870fb0abc1bda0bbb160565151a29"} Jul 31 19:55:42.613167 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:42.613146 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tncq5" event={"ID":"84ef4519-773b-47ba-b51f-aaf05966fd13","Type":"ContainerStarted","Data":"1c096cf24e8c7b06ce8e5d50a2e6790597c0d2b12732545862cc05feee17cc47"} Jul 31 19:55:42.613292 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:42.613277 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:55:42.625900 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:42.625860 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hf82d" podStartSLOduration=67.082002405 podStartE2EDuration="1m11.625846698s" totalImagesPullingTime="4.543844293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:42.624759839 +0000 UTC m=+71.966343461" watchObservedRunningTime="2026-07-31 19:55:42.625846698 +0000 UTC m=+71.967430323" Jul 31 19:55:42.638475 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:42.638435 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tncq5" podStartSLOduration=67.113503152 podStartE2EDuration="1m11.63842184s" totalImagesPullingTime="4.524918688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:42.637934021 +0000 UTC m=+71.979517652" watchObservedRunningTime="2026-07-31 19:55:42.63842184 +0000 UTC m=+71.980005465" Jul 31 19:55:44.828183 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:44.828149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:44.828183 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:44.828186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:44.828573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:44.828264 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 19:55:44.828573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:44.828289 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 19:55:44.828573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:44.828339 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:52.828322283 +0000 UTC m=+82.169905903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : configmap "kube-rbac-proxy" not found Jul 31 19:55:44.828573 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:55:44.828353 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls podName:81566aae-bac4-44f8-b062-5e88eeaa90a3 nodeName:}" failed. No retries permitted until 2026-07-31 19:55:52.828347613 +0000 UTC m=+82.169931217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqrm2" (UID: "81566aae-bac4-44f8-b062-5e88eeaa90a3") : secret "insights-runtime-extractor-tls" not found Jul 31 19:55:52.888704 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:52.888657 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:52.888704 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:52.888703 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:52.889206 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:52.889186 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81566aae-bac4-44f8-b062-5e88eeaa90a3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:52.891038 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:52.891015 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81566aae-bac4-44f8-b062-5e88eeaa90a3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqrm2\" (UID: \"81566aae-bac4-44f8-b062-5e88eeaa90a3\") " pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:53.012511 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:53.012483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqrm2" Jul 31 19:55:53.133574 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:53.131772 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqrm2"] Jul 31 19:55:53.134410 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:55:53.134368 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81566aae_bac4_44f8_b062_5e88eeaa90a3.slice/crio-979d46218313b28373755c55e751b975bca67627c3830a1a72871c6cd9b78421 WatchSource:0}: Error finding container 979d46218313b28373755c55e751b975bca67627c3830a1a72871c6cd9b78421: Status 404 returned error can't find the container with id 979d46218313b28373755c55e751b975bca67627c3830a1a72871c6cd9b78421 Jul 31 19:55:53.647665 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:53.647632 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqrm2" event={"ID":"81566aae-bac4-44f8-b062-5e88eeaa90a3","Type":"ContainerStarted","Data":"b57498b1c91f351f101a9f409ae7887d087a0a1ef8ab5a60df3a6cea9661df33"} Jul 31 19:55:53.647820 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:53.647694 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqrm2" event={"ID":"81566aae-bac4-44f8-b062-5e88eeaa90a3","Type":"ContainerStarted","Data":"979d46218313b28373755c55e751b975bca67627c3830a1a72871c6cd9b78421"} Jul 31 19:55:54.653369 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:54.653332 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqrm2" event={"ID":"81566aae-bac4-44f8-b062-5e88eeaa90a3","Type":"ContainerStarted","Data":"b4b31b1a53c49f126ad6d3a633e3f7a62e9a170d7609c913667314255d30f550"} Jul 31 19:55:55.659759 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:55.659672 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqrm2" event={"ID":"81566aae-bac4-44f8-b062-5e88eeaa90a3","Type":"ContainerStarted","Data":"32819b4e3c2270642beddca73f53a6810db257db670d56f49e30813e78eecd19"} Jul 31 19:55:55.677398 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:55:55.677350 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gqrm2" podStartSLOduration=16.593336213 podStartE2EDuration="18.677338095s" totalImagesPullingTime="2.084001882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:55:55.676345403 +0000 UTC m=+85.017929028" watchObservedRunningTime="2026-07-31 19:55:55.677338095 +0000 UTC m=+85.018921715" Jul 31 19:56:09.005023 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.004968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:09.007358 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.007335 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"image-registry-6d6457b674-ntd74\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:09.105375 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.105335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:56:09.107598 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.107574 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57414e35-918a-41db-a3ad-cd04428a36dd-metrics-tls\") pod \"dns-default-48wns\" (UID: \"57414e35-918a-41db-a3ad-cd04428a36dd\") " pod="openshift-dns/dns-default-48wns" Jul 31 19:56:09.264694 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.264616 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dg6cw\"" Jul 31 19:56:09.272546 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.272530 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:09.313557 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.313532 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6742\"" Jul 31 19:56:09.321697 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.321672 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48wns" Jul 31 19:56:09.423932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.423886 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:56:09.426179 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:56:09.426153 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390a3ac8_4eac_4f11_8c5b_25b9a5c3eb96.slice/crio-c5a0ff993e06812a2ac6d5d34ccf9c55ac4a52414191ba7a61ec67c27e890de8 WatchSource:0}: Error finding container c5a0ff993e06812a2ac6d5d34ccf9c55ac4a52414191ba7a61ec67c27e890de8: Status 404 returned error can't find the container with id c5a0ff993e06812a2ac6d5d34ccf9c55ac4a52414191ba7a61ec67c27e890de8 Jul 31 19:56:09.448187 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.448135 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48wns"] Jul 31 19:56:09.452386 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:56:09.452364 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57414e35_918a_41db_a3ad_cd04428a36dd.slice/crio-09d8dafcc4cfafa6014ca5591fd94cc0a9d0a471cc00d97a2fb9413133b16f80 WatchSource:0}: Error finding container 09d8dafcc4cfafa6014ca5591fd94cc0a9d0a471cc00d97a2fb9413133b16f80: Status 404 returned error can't find the container with id 09d8dafcc4cfafa6014ca5591fd94cc0a9d0a471cc00d97a2fb9413133b16f80 Jul 31 19:56:09.704916 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.704876 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wns" event={"ID":"57414e35-918a-41db-a3ad-cd04428a36dd","Type":"ContainerStarted","Data":"09d8dafcc4cfafa6014ca5591fd94cc0a9d0a471cc00d97a2fb9413133b16f80"} Jul 31 19:56:09.706286 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.706259 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" event={"ID":"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96","Type":"ContainerStarted","Data":"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96"} Jul 31 19:56:09.706415 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.706303 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" event={"ID":"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96","Type":"ContainerStarted","Data":"c5a0ff993e06812a2ac6d5d34ccf9c55ac4a52414191ba7a61ec67c27e890de8"} Jul 31 19:56:09.706415 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.706358 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:09.726226 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:09.726187 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" podStartSLOduration=98.726173729 podStartE2EDuration="1m38.726173729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:56:09.72494285 +0000 UTC m=+99.066526476" watchObservedRunningTime="2026-07-31 19:56:09.726173729 +0000 UTC m=+99.067757354" Jul 31 19:56:11.713812 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:11.713772 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wns" event={"ID":"57414e35-918a-41db-a3ad-cd04428a36dd","Type":"ContainerStarted","Data":"4c9e54c410c98f7b7ff0493865c54da98fe59c33834d767f2cfa5a4f9ccbb441"} Jul 31 19:56:11.714201 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:11.713833 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48wns" event={"ID":"57414e35-918a-41db-a3ad-cd04428a36dd","Type":"ContainerStarted","Data":"80f0a0c92eb212876edd5fc002eea91ac54dd7504d1c3e4f95ace8db604d8858"} Jul 31 19:56:11.714201 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:11.713892 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-48wns" Jul 31 19:56:11.731049 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:11.731005 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-48wns" podStartSLOduration=64.947929618 podStartE2EDuration="1m6.730992596s" totalImagesPullingTime="1.783062978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:56:11.730051404 +0000 UTC m=+101.071635030" watchObservedRunningTime="2026-07-31 19:56:11.730992596 +0000 UTC m=+101.072576222" Jul 31 19:56:13.618553 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:13.618526 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tncq5" Jul 31 19:56:15.578425 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.578391 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5hrxj"] Jul 31 19:56:15.610079 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.610054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.612364 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.612341 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 19:56:15.612705 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.612686 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 19:56:15.612811 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.612707 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j8t45\"" Jul 31 19:56:15.613346 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.613328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 19:56:15.613391 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.613330 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 19:56:15.613597 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.613582 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 19:56:15.613638 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.613610 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 19:56:15.750190 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750407 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750224 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750407 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-wtmp\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750407 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750294 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750407 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750368 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-root\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750407 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750390 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-metrics-client-ca\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750613 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzq4z\" (UniqueName: \"kubernetes.io/projected/ceabba42-5a1a-4eb3-8d94-6cea37e46571-kube-api-access-pzq4z\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750613 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-sys\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.750613 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.750533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-textfile\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851650 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-root\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851650 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-metrics-client-ca\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851650 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzq4z\" (UniqueName: \"kubernetes.io/projected/ceabba42-5a1a-4eb3-8d94-6cea37e46571-kube-api-access-pzq4z\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851650 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-sys\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851665 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-textfile\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-root\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-sys\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:56:15.851792 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-wtmp\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:56:15.851845 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls podName:ceabba42-5a1a-4eb3-8d94-6cea37e46571 nodeName:}" failed. No retries permitted until 2026-07-31 19:56:16.351827594 +0000 UTC m=+105.693411201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls") pod "node-exporter-5hrxj" (UID: "ceabba42-5a1a-4eb3-8d94-6cea37e46571") : secret "node-exporter-tls" not found Jul 31 19:56:15.851941 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.852284 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.851956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-wtmp\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.852284 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.852078 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-textfile\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.852284 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.852198 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-metrics-client-ca\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.852410 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.852292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-accelerators-collector-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.853945 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.853928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:15.861264 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:15.861243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzq4z\" (UniqueName: \"kubernetes.io/projected/ceabba42-5a1a-4eb3-8d94-6cea37e46571-kube-api-access-pzq4z\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:16.356142 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:16.356108 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:16.358264 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:16.358243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ceabba42-5a1a-4eb3-8d94-6cea37e46571-node-exporter-tls\") pod \"node-exporter-5hrxj\" (UID: \"ceabba42-5a1a-4eb3-8d94-6cea37e46571\") " pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:16.519463 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:16.519427 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5hrxj" Jul 31 19:56:16.529262 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:56:16.529231 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceabba42_5a1a_4eb3_8d94_6cea37e46571.slice/crio-c40c3999b37304e13102cfee39a7fe9fbcd2920448641a782f956bc02e9e55b6 WatchSource:0}: Error finding container c40c3999b37304e13102cfee39a7fe9fbcd2920448641a782f956bc02e9e55b6: Status 404 returned error can't find the container with id c40c3999b37304e13102cfee39a7fe9fbcd2920448641a782f956bc02e9e55b6 Jul 31 19:56:16.730405 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:16.730298 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hrxj" event={"ID":"ceabba42-5a1a-4eb3-8d94-6cea37e46571","Type":"ContainerStarted","Data":"c40c3999b37304e13102cfee39a7fe9fbcd2920448641a782f956bc02e9e55b6"} Jul 31 19:56:17.734755 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:17.734721 2556 generic.go:356] "Generic (PLEG): container finished" podID="ceabba42-5a1a-4eb3-8d94-6cea37e46571" containerID="14bb2a5ff243293a67da9df6cbb6e0faf1803c85def77040db8d578e8fa8f385" exitCode=0 Jul 31 19:56:17.735147 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:17.734767 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hrxj" event={"ID":"ceabba42-5a1a-4eb3-8d94-6cea37e46571","Type":"ContainerDied","Data":"14bb2a5ff243293a67da9df6cbb6e0faf1803c85def77040db8d578e8fa8f385"} Jul 31 19:56:18.739639 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:18.739604 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hrxj" event={"ID":"ceabba42-5a1a-4eb3-8d94-6cea37e46571","Type":"ContainerStarted","Data":"536ac546a8e61377ab4d6704f2cad8655789c3e15d7375ff737d2c99fa2405f9"} Jul 31 19:56:18.739996 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:18.739657 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5hrxj" event={"ID":"ceabba42-5a1a-4eb3-8d94-6cea37e46571","Type":"ContainerStarted","Data":"1f929b5e479de9617035dd76c8b6171f546c6ed45086b346552055f9811f1338"} Jul 31 19:56:18.757612 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:18.757567 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5hrxj" podStartSLOduration=2.7008991780000002 podStartE2EDuration="3.757552884s" totalImagesPullingTime="996.79697ms" totalInitContainerRuntime="59.856736ms" isStatefulPod=true podCreationTimestamp="2026-07-31 19:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:56:18.756355257 +0000 UTC m=+108.097938901" watchObservedRunningTime="2026-07-31 19:56:18.757552884 +0000 UTC m=+108.099136509" Jul 31 19:56:21.719514 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:21.719483 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-48wns" Jul 31 19:56:29.277441 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:29.277408 2556 patch_prober.go:28] interesting pod/image-registry-6d6457b674-ntd74 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 31 19:56:29.277837 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:29.277472 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:56:30.715260 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:30.715234 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:30.938850 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:30.938816 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:56:40.932902 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:40.932863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:56:40.935122 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:40.935102 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a481939-8e1f-4acd-9361-e7095e8d52f3-metrics-certs\") pod \"network-metrics-daemon-qn4bd\" (UID: \"7a481939-8e1f-4acd-9361-e7095e8d52f3\") " pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:56:40.975110 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:40.975084 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twv2l\"" Jul 31 19:56:40.982959 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:40.982938 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qn4bd" Jul 31 19:56:41.102811 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:41.102781 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qn4bd"] Jul 31 19:56:41.105826 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:56:41.105799 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a481939_8e1f_4acd_9361_e7095e8d52f3.slice/crio-bd3fe31cbc8b51a6bda25919ea773afd617cb481c0b3acffff03afd38c9590e8 WatchSource:0}: Error finding container bd3fe31cbc8b51a6bda25919ea773afd617cb481c0b3acffff03afd38c9590e8: Status 404 returned error can't find the container with id bd3fe31cbc8b51a6bda25919ea773afd617cb481c0b3acffff03afd38c9590e8 Jul 31 19:56:41.818057 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:41.818019 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qn4bd" event={"ID":"7a481939-8e1f-4acd-9361-e7095e8d52f3","Type":"ContainerStarted","Data":"bd3fe31cbc8b51a6bda25919ea773afd617cb481c0b3acffff03afd38c9590e8"} Jul 31 19:56:42.823201 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:42.823164 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qn4bd" event={"ID":"7a481939-8e1f-4acd-9361-e7095e8d52f3","Type":"ContainerStarted","Data":"f8d033b3af40496aa647b2ba1a61d907ae92512a5e1b28f67f5464186fd5d4ca"} Jul 31 19:56:42.823671 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:42.823227 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qn4bd" event={"ID":"7a481939-8e1f-4acd-9361-e7095e8d52f3","Type":"ContainerStarted","Data":"f7d72bdd481547d46be9935886f8a5cc9879be1a726410fe1dc33ccefc5c2aa7"} Jul 31 19:56:42.838863 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:42.838817 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qn4bd" podStartSLOduration=130.692930002 podStartE2EDuration="2m11.838805279s" totalImagesPullingTime="1.145875277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:56:42.83832906 +0000 UTC m=+132.179912678" watchObservedRunningTime="2026-07-31 19:56:42.838805279 +0000 UTC m=+132.180388905" Jul 31 19:56:45.394955 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:45.394919 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" podUID="c2ae9309-4896-448f-b68d-0bbcb57bd552" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 19:56:55.394961 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:55.394920 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" podUID="c2ae9309-4896-448f-b68d-0bbcb57bd552" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 19:56:55.962138 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:55.962062 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerName="registry" containerID="cri-o://fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96" gracePeriod=30 Jul 31 19:56:56.196431 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.196403 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:56.242458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242390 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242430 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242449 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgb7p\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242709 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242590 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242709 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242644 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242709 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242692 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242865 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242737 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.242865 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.242772 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates\") pod \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\" (UID: \"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96\") " Jul 31 19:56:56.243077 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.243051 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:56:56.243351 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.243325 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:56:56.244776 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.244752 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:56:56.244877 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.244787 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:56:56.244959 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.244935 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p" (OuterVolumeSpecName: "kube-api-access-wgb7p") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "kube-api-access-wgb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:56:56.245014 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.244965 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:56:56.245208 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.245183 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:56:56.251344 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.251303 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" (UID: "390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:56:56.343611 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343590 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-tls\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343611 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343611 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-bound-sa-token\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343620 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgb7p\" (UniqueName: \"kubernetes.io/projected/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-kube-api-access-wgb7p\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343631 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-image-registry-private-configuration\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343639 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-trusted-ca\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343647 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-ca-trust-extracted\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343655 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-installation-pull-secrets\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.343734 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.343663 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96-registry-certificates\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:56:56.869652 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.869620 2556 generic.go:356] "Generic (PLEG): container finished" podID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerID="fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96" exitCode=0 Jul 31 19:56:56.870067 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.869662 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" Jul 31 19:56:56.870067 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.869678 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" event={"ID":"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96","Type":"ContainerDied","Data":"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96"} Jul 31 19:56:56.870067 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.869719 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6457b674-ntd74" event={"ID":"390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96","Type":"ContainerDied","Data":"c5a0ff993e06812a2ac6d5d34ccf9c55ac4a52414191ba7a61ec67c27e890de8"} Jul 31 19:56:56.870067 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.869738 2556 scope.go:117] "RemoveContainer" containerID="fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96" Jul 31 19:56:56.877929 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.877911 2556 scope.go:117] "RemoveContainer" containerID="fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96" Jul 31 19:56:56.878177 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:56:56.878154 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96\": container with ID starting with fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96 not found: ID does not exist" containerID="fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96" Jul 31 19:56:56.878224 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.878184 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96"} err="failed to get container status \"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96\": rpc error: code = NotFound desc = could not find container \"fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96\": container with ID starting with fed15f79c64aa3eb6bc9ebd231586ba8889f723a64712e0f0ab99d0432354f96 not found: ID does not exist" Jul 31 19:56:56.898220 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.898187 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:56:56.902028 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:56.902002 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d6457b674-ntd74"] Jul 31 19:56:57.164725 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:56:57.164641 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" path="/var/lib/kubelet/pods/390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96/volumes" Jul 31 19:57:05.394855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.394817 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" podUID="c2ae9309-4896-448f-b68d-0bbcb57bd552" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 19:57:05.395204 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.394888 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" Jul 31 19:57:05.395354 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.395335 2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6c1d569f70cc14343bfc11ee47beb58d0c5e92ce4c0b2da9afa023f200290c78"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 31 19:57:05.395395 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.395373 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" podUID="c2ae9309-4896-448f-b68d-0bbcb57bd552" containerName="service-proxy" containerID="cri-o://6c1d569f70cc14343bfc11ee47beb58d0c5e92ce4c0b2da9afa023f200290c78" gracePeriod=30 Jul 31 19:57:05.903022 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.902987 2556 generic.go:356] "Generic (PLEG): container finished" podID="c2ae9309-4896-448f-b68d-0bbcb57bd552" containerID="6c1d569f70cc14343bfc11ee47beb58d0c5e92ce4c0b2da9afa023f200290c78" exitCode=2 Jul 31 19:57:05.903198 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.903058 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerDied","Data":"6c1d569f70cc14343bfc11ee47beb58d0c5e92ce4c0b2da9afa023f200290c78"} Jul 31 19:57:05.903198 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:57:05.903106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78bb7597cf-425z2" event={"ID":"c2ae9309-4896-448f-b68d-0bbcb57bd552","Type":"ContainerStarted","Data":"ec088be8daf753e31d53c79b76b949be5947eeda8934b01ab938947cf5ce928a"} Jul 31 19:58:18.754127 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.754097 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt"] Jul 31 19:58:18.754555 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.754332 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerName="registry" Jul 31 19:58:18.754555 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.754345 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerName="registry" Jul 31 19:58:18.754555 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.754399 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="390a3ac8-4eac-4f11-8c5b-25b9a5c3eb96" containerName="registry" Jul 31 19:58:18.757305 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.757289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.759561 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.759537 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 19:58:18.759683 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.759642 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 19:58:18.760409 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.760394 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nbqwr\"" Jul 31 19:58:18.765512 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.765490 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt"] Jul 31 19:58:18.873069 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.873040 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.873069 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.873072 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.873245 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.873106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6sf\" (UniqueName: \"kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.973617 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.973594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.973714 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.973632 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6sf\" (UniqueName: \"kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.973714 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.973663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.973964 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.973950 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.974012 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.973981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:18.981754 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:18.981733 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6sf\" (UniqueName: \"kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:19.066989 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:19.066967 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:19.186259 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:19.186138 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt"] Jul 31 19:58:19.188961 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:58:19.188937 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25339c51_119f_495a_8e80_1a31cefc6703.slice/crio-9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986 WatchSource:0}: Error finding container 9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986: Status 404 returned error can't find the container with id 9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986 Jul 31 19:58:20.135056 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:20.135022 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" event={"ID":"25339c51-119f-495a-8e80-1a31cefc6703","Type":"ContainerStarted","Data":"9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986"} Jul 31 19:58:26.156051 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:26.156016 2556 generic.go:356] "Generic (PLEG): container finished" podID="25339c51-119f-495a-8e80-1a31cefc6703" containerID="92c7570333004363cb8deee8df37721a79c879e5a1cc4a735d17fbee86dd5ee3" exitCode=0 Jul 31 19:58:26.156430 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:26.156068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" event={"ID":"25339c51-119f-495a-8e80-1a31cefc6703","Type":"ContainerDied","Data":"92c7570333004363cb8deee8df37721a79c879e5a1cc4a735d17fbee86dd5ee3"} Jul 31 19:58:29.168290 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:29.168257 2556 generic.go:356] "Generic (PLEG): container finished" podID="25339c51-119f-495a-8e80-1a31cefc6703" containerID="c38615ad68869bd0a0dfb93755f8577a8eb826173306c94cc598b49a61d5132f" exitCode=0 Jul 31 19:58:29.168674 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:29.168340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" event={"ID":"25339c51-119f-495a-8e80-1a31cefc6703","Type":"ContainerDied","Data":"c38615ad68869bd0a0dfb93755f8577a8eb826173306c94cc598b49a61d5132f"} Jul 31 19:58:36.198608 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:36.198572 2556 generic.go:356] "Generic (PLEG): container finished" podID="25339c51-119f-495a-8e80-1a31cefc6703" containerID="0db5fab089b0c14e5ff4151ba1b22477d112a8bb50af7ba077ce2856122482f5" exitCode=0 Jul 31 19:58:36.199030 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:36.198633 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" event={"ID":"25339c51-119f-495a-8e80-1a31cefc6703","Type":"ContainerDied","Data":"0db5fab089b0c14e5ff4151ba1b22477d112a8bb50af7ba077ce2856122482f5"} Jul 31 19:58:37.312563 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.312536 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:37.400071 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.400036 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m6sf\" (UniqueName: \"kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf\") pod \"25339c51-119f-495a-8e80-1a31cefc6703\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " Jul 31 19:58:37.400252 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.400089 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle\") pod \"25339c51-119f-495a-8e80-1a31cefc6703\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " Jul 31 19:58:37.400252 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.400118 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util\") pod \"25339c51-119f-495a-8e80-1a31cefc6703\" (UID: \"25339c51-119f-495a-8e80-1a31cefc6703\") " Jul 31 19:58:37.400826 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.400801 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle" (OuterVolumeSpecName: "bundle") pod "25339c51-119f-495a-8e80-1a31cefc6703" (UID: "25339c51-119f-495a-8e80-1a31cefc6703"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:58:37.402104 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.402085 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf" (OuterVolumeSpecName: "kube-api-access-8m6sf") pod "25339c51-119f-495a-8e80-1a31cefc6703" (UID: "25339c51-119f-495a-8e80-1a31cefc6703"). InnerVolumeSpecName "kube-api-access-8m6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:58:37.403999 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.403981 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util" (OuterVolumeSpecName: "util") pod "25339c51-119f-495a-8e80-1a31cefc6703" (UID: "25339c51-119f-495a-8e80-1a31cefc6703"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:58:37.500637 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.500558 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:37.500637 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.500586 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6sf\" (UniqueName: \"kubernetes.io/projected/25339c51-119f-495a-8e80-1a31cefc6703-kube-api-access-8m6sf\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:37.500637 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:37.500596 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25339c51-119f-495a-8e80-1a31cefc6703-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:38.205708 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:38.205676 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" Jul 31 19:58:38.205875 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:38.205676 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754079hnt" event={"ID":"25339c51-119f-495a-8e80-1a31cefc6703","Type":"ContainerDied","Data":"9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986"} Jul 31 19:58:38.205875 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:38.205797 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b051ad79c6cb7b673af649b7323d1a25eda1f3808e6e472a632cfe066ddf986" Jul 31 19:58:45.607482 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607453 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2"] Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607658 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="util" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607667 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="util" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607680 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="extract" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607686 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="extract" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607694 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="pull" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607699 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="pull" Jul 31 19:58:45.607936 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.607735 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="25339c51-119f-495a-8e80-1a31cefc6703" containerName="extract" Jul 31 19:58:45.658974 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.658940 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2"] Jul 31 19:58:45.659111 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.659056 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.661459 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.661436 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 19:58:45.661459 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.661449 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 19:58:45.662252 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.662239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nbqwr\"" Jul 31 19:58:45.745573 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.745542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.745573 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.745573 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs4rq\" (UniqueName: \"kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.745725 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.745626 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.846669 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.846641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.846786 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.846678 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.846786 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.846697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs4rq\" (UniqueName: \"kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.847031 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.847012 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.847067 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.847035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.854357 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.854327 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs4rq\" (UniqueName: \"kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:45.968240 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:45.968157 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:46.114910 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:46.114876 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2"] Jul 31 19:58:46.118234 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:58:46.118197 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37314a07_28ae_4862_bc5f_396d29225212.slice/crio-47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180 WatchSource:0}: Error finding container 47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180: Status 404 returned error can't find the container with id 47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180 Jul 31 19:58:46.232385 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:46.232286 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerStarted","Data":"cbc7a54d4dfa7cbf698b8fdcdcf70320a5f9dfd67ba2a9b7bef65c0f42b39507"} Jul 31 19:58:46.232385 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:46.232355 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerStarted","Data":"47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180"} Jul 31 19:58:47.236722 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:47.236640 2556 generic.go:356] "Generic (PLEG): container finished" podID="37314a07-28ae-4862-bc5f-396d29225212" containerID="cbc7a54d4dfa7cbf698b8fdcdcf70320a5f9dfd67ba2a9b7bef65c0f42b39507" exitCode=0 Jul 31 19:58:47.236722 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:47.236681 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerDied","Data":"cbc7a54d4dfa7cbf698b8fdcdcf70320a5f9dfd67ba2a9b7bef65c0f42b39507"} Jul 31 19:58:49.246155 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:49.246129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerStarted","Data":"9a0ebee6bef830e944e608fdf4addd63de319be7ac37305542fb6fc083f7fceb"} Jul 31 19:58:50.250555 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:50.250516 2556 generic.go:356] "Generic (PLEG): container finished" podID="37314a07-28ae-4862-bc5f-396d29225212" containerID="9a0ebee6bef830e944e608fdf4addd63de319be7ac37305542fb6fc083f7fceb" exitCode=0 Jul 31 19:58:50.250555 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:50.250556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerDied","Data":"9a0ebee6bef830e944e608fdf4addd63de319be7ac37305542fb6fc083f7fceb"} Jul 31 19:58:51.255834 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:51.255804 2556 generic.go:356] "Generic (PLEG): container finished" podID="37314a07-28ae-4862-bc5f-396d29225212" containerID="b381eb54add66a73af1dc9da74a88dc77f6971dbce11f3b0aafde960572f8c02" exitCode=0 Jul 31 19:58:51.256195 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:51.255878 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerDied","Data":"b381eb54add66a73af1dc9da74a88dc77f6971dbce11f3b0aafde960572f8c02"} Jul 31 19:58:52.375540 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.375518 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:52.386456 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.386438 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle\") pod \"37314a07-28ae-4862-bc5f-396d29225212\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " Jul 31 19:58:52.386576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.386478 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs4rq\" (UniqueName: \"kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq\") pod \"37314a07-28ae-4862-bc5f-396d29225212\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " Jul 31 19:58:52.386576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.386525 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util\") pod \"37314a07-28ae-4862-bc5f-396d29225212\" (UID: \"37314a07-28ae-4862-bc5f-396d29225212\") " Jul 31 19:58:52.386850 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.386828 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle" (OuterVolumeSpecName: "bundle") pod "37314a07-28ae-4862-bc5f-396d29225212" (UID: "37314a07-28ae-4862-bc5f-396d29225212"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:58:52.388371 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.388353 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq" (OuterVolumeSpecName: "kube-api-access-qs4rq") pod "37314a07-28ae-4862-bc5f-396d29225212" (UID: "37314a07-28ae-4862-bc5f-396d29225212"). InnerVolumeSpecName "kube-api-access-qs4rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:58:52.393018 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.392981 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util" (OuterVolumeSpecName: "util") pod "37314a07-28ae-4862-bc5f-396d29225212" (UID: "37314a07-28ae-4862-bc5f-396d29225212"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:58:52.486916 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.486894 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:52.486916 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.486914 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37314a07-28ae-4862-bc5f-396d29225212-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:52.486916 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:52.486923 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs4rq\" (UniqueName: \"kubernetes.io/projected/37314a07-28ae-4862-bc5f-396d29225212-kube-api-access-qs4rq\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:58:53.264457 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:53.264429 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" Jul 31 19:58:53.264457 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:53.264438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx6bq2" event={"ID":"37314a07-28ae-4862-bc5f-396d29225212","Type":"ContainerDied","Data":"47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180"} Jul 31 19:58:53.264654 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:53.264486 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47746d3ff7aaffaa78f407cf485ed07f02e7afac16b836e981af10671fc63180" Jul 31 19:58:56.106496 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106456 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n8t5d"] Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106695 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="pull" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106709 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="pull" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106724 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="extract" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106730 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="extract" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106738 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="util" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106743 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="util" Jul 31 19:58:56.106855 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.106791 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="37314a07-28ae-4862-bc5f-396d29225212" containerName="extract" Jul 31 19:58:56.109505 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.109488 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.111808 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.111779 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 19:58:56.111808 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.111788 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 19:58:56.112632 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.112618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lk7zf\"" Jul 31 19:58:56.117124 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.117103 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n8t5d"] Jul 31 19:58:56.209602 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.209569 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-bound-sa-token\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.209763 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.209664 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kt6b\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-kube-api-access-8kt6b\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.310877 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.310849 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kt6b\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-kube-api-access-8kt6b\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.311023 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.310888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-bound-sa-token\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.319239 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.319210 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-bound-sa-token\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.319359 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.319267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kt6b\" (UniqueName: \"kubernetes.io/projected/618aa95b-a0e7-4785-bcaa-c8220710e404-kube-api-access-8kt6b\") pod \"cert-manager-55b4d954d5-n8t5d\" (UID: \"618aa95b-a0e7-4785-bcaa-c8220710e404\") " pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.419065 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.419005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-n8t5d" Jul 31 19:58:56.541538 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:56.541437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n8t5d"] Jul 31 19:58:56.543977 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:58:56.543948 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod618aa95b_a0e7_4785_bcaa_c8220710e404.slice/crio-b94262bab79a2eca30b601a428fd0a5729e8643b5d1ed88278a3700b73b8b164 WatchSource:0}: Error finding container b94262bab79a2eca30b601a428fd0a5729e8643b5d1ed88278a3700b73b8b164: Status 404 returned error can't find the container with id b94262bab79a2eca30b601a428fd0a5729e8643b5d1ed88278a3700b73b8b164 Jul 31 19:58:57.278237 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:57.278202 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-n8t5d" event={"ID":"618aa95b-a0e7-4785-bcaa-c8220710e404","Type":"ContainerStarted","Data":"b94262bab79a2eca30b601a428fd0a5729e8643b5d1ed88278a3700b73b8b164"} Jul 31 19:58:59.286944 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:59.286863 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-n8t5d" event={"ID":"618aa95b-a0e7-4785-bcaa-c8220710e404","Type":"ContainerStarted","Data":"f1f733909acbd5850de102de7b489f3ac4507096c15221548c0f303fff62c462"} Jul 31 19:58:59.302121 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:58:59.302068 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-n8t5d" podStartSLOduration=0.840815878 podStartE2EDuration="3.302052486s" totalImagesPullingTime="2.461236608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:58:59.301416708 +0000 UTC m=+268.643000333" watchObservedRunningTime="2026-07-31 19:58:59.302052486 +0000 UTC m=+268.643636112" Jul 31 19:59:11.609581 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.609547 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x"] Jul 31 19:59:11.611956 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.611939 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.614185 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.614159 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 19:59:11.614915 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.614901 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 19:59:11.614915 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.614909 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nbqwr\"" Jul 31 19:59:11.620023 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.620001 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x"] Jul 31 19:59:11.710398 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.710365 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.710505 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.710407 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.710505 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.710445 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585f6\" (UniqueName: \"kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.811426 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.811389 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585f6\" (UniqueName: \"kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.811561 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.811452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.811561 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.811488 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.811823 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.811806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.811872 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.811856 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.820932 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.820907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-585f6\" (UniqueName: \"kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:11.920975 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:11.920911 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:12.038196 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:12.038167 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x"] Jul 31 19:59:12.040977 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:12.040946 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d43f5e_14ca_4a39_bd8d_666b9db05cea.slice/crio-da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671 WatchSource:0}: Error finding container da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671: Status 404 returned error can't find the container with id da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671 Jul 31 19:59:12.335094 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:12.335060 2556 generic.go:356] "Generic (PLEG): container finished" podID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerID="1657d456cf28e55ce6d5060b05ba5e0f2a1e56ee668ae9e4fb393b60efe625c6" exitCode=0 Jul 31 19:59:12.335256 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:12.335148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" event={"ID":"87d43f5e-14ca-4a39-bd8d-666b9db05cea","Type":"ContainerDied","Data":"1657d456cf28e55ce6d5060b05ba5e0f2a1e56ee668ae9e4fb393b60efe625c6"} Jul 31 19:59:12.335256 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:12.335201 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" event={"ID":"87d43f5e-14ca-4a39-bd8d-666b9db05cea","Type":"ContainerStarted","Data":"da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671"} Jul 31 19:59:13.340486 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:13.340449 2556 generic.go:356] "Generic (PLEG): container finished" podID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerID="acad748c760c7cf72875cf2cd19ba13b57a6ddd3039430fe370c73a4368bd984" exitCode=0 Jul 31 19:59:13.340841 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:13.340512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" event={"ID":"87d43f5e-14ca-4a39-bd8d-666b9db05cea","Type":"ContainerDied","Data":"acad748c760c7cf72875cf2cd19ba13b57a6ddd3039430fe370c73a4368bd984"} Jul 31 19:59:14.345784 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:14.345753 2556 generic.go:356] "Generic (PLEG): container finished" podID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerID="90820a883339f0508d4e5068737fdc6dc2df253a5c79e52410f059766cedbff7" exitCode=0 Jul 31 19:59:14.346141 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:14.345802 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" event={"ID":"87d43f5e-14ca-4a39-bd8d-666b9db05cea","Type":"ContainerDied","Data":"90820a883339f0508d4e5068737fdc6dc2df253a5c79e52410f059766cedbff7"} Jul 31 19:59:15.466578 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.466553 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:15.535814 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.535783 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-585f6\" (UniqueName: \"kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6\") pod \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " Jul 31 19:59:15.535940 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.535851 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle\") pod \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " Jul 31 19:59:15.535940 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.535883 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util\") pod \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\" (UID: \"87d43f5e-14ca-4a39-bd8d-666b9db05cea\") " Jul 31 19:59:15.536712 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.536678 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle" (OuterVolumeSpecName: "bundle") pod "87d43f5e-14ca-4a39-bd8d-666b9db05cea" (UID: "87d43f5e-14ca-4a39-bd8d-666b9db05cea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:59:15.537624 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.537602 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6" (OuterVolumeSpecName: "kube-api-access-585f6") pod "87d43f5e-14ca-4a39-bd8d-666b9db05cea" (UID: "87d43f5e-14ca-4a39-bd8d-666b9db05cea"). InnerVolumeSpecName "kube-api-access-585f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:59:15.541448 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.541426 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util" (OuterVolumeSpecName: "util") pod "87d43f5e-14ca-4a39-bd8d-666b9db05cea" (UID: "87d43f5e-14ca-4a39-bd8d-666b9db05cea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:59:15.637149 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.637068 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-585f6\" (UniqueName: \"kubernetes.io/projected/87d43f5e-14ca-4a39-bd8d-666b9db05cea-kube-api-access-585f6\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:15.637149 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.637107 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:15.637149 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:15.637116 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87d43f5e-14ca-4a39-bd8d-666b9db05cea-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:16.354030 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:16.353994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" event={"ID":"87d43f5e-14ca-4a39-bd8d-666b9db05cea","Type":"ContainerDied","Data":"da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671"} Jul 31 19:59:16.354030 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:16.354025 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4ts9x" Jul 31 19:59:16.354301 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:16.354042 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da534a10e60db0c143eddae8d4f771dd20a3da3166dcada97d2d58c9ee1de671" Jul 31 19:59:23.669848 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.669814 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc"] Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670031 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="extract" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670042 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="extract" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670061 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="pull" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670066 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="pull" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670075 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="util" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670079 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="util" Jul 31 19:59:23.670213 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.670117 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d43f5e-14ca-4a39-bd8d-666b9db05cea" containerName="extract" Jul 31 19:59:23.677452 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.677431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.680385 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.680357 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 19:59:23.695232 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.695210 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 19:59:23.695511 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.695491 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 19:59:23.695590 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.695544 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 19:59:23.695638 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.695610 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ff9cq\"" Jul 31 19:59:23.703724 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.703705 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc"] Jul 31 19:59:23.793449 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.793414 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-tmp\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.793449 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.793451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmn9\" (UniqueName: \"kubernetes.io/projected/c26f3a50-49d3-47df-bd54-bfb9d76361fe-kube-api-access-hbmn9\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.793661 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.793470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.793661 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.793491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.793661 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.793551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-manifests\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.894719 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.894686 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-manifests\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.894891 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.894736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-tmp\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.894891 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.894808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmn9\" (UniqueName: \"kubernetes.io/projected/c26f3a50-49d3-47df-bd54-bfb9d76361fe-kube-api-access-hbmn9\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.894891 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.894825 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.894891 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.894845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.895165 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.895123 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-manifests\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.897060 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.897031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c26f3a50-49d3-47df-bd54-bfb9d76361fe-tmp\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.897264 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.897248 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.897336 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.897300 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c26f3a50-49d3-47df-bd54-bfb9d76361fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.903639 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.903618 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmn9\" (UniqueName: \"kubernetes.io/projected/c26f3a50-49d3-47df-bd54-bfb9d76361fe-kube-api-access-hbmn9\") pod \"opendatahub-operator-controller-manager-675f995665-8s4gc\" (UID: \"c26f3a50-49d3-47df-bd54-bfb9d76361fe\") " pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:23.986640 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:23.986563 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:24.110128 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:24.110099 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc"] Jul 31 19:59:24.113095 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:24.113072 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26f3a50_49d3_47df_bd54_bfb9d76361fe.slice/crio-124c2ebf67e599c5b9e74c137b4583688abc28252afdfd481d1c2ef677d7dd88 WatchSource:0}: Error finding container 124c2ebf67e599c5b9e74c137b4583688abc28252afdfd481d1c2ef677d7dd88: Status 404 returned error can't find the container with id 124c2ebf67e599c5b9e74c137b4583688abc28252afdfd481d1c2ef677d7dd88 Jul 31 19:59:24.385382 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:24.385346 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" event={"ID":"c26f3a50-49d3-47df-bd54-bfb9d76361fe","Type":"ContainerStarted","Data":"124c2ebf67e599c5b9e74c137b4583688abc28252afdfd481d1c2ef677d7dd88"} Jul 31 19:59:27.402867 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:27.402835 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" event={"ID":"c26f3a50-49d3-47df-bd54-bfb9d76361fe","Type":"ContainerStarted","Data":"142a651b4268f69a42674d99c4431ba2871b6d4c5a1297172dd7342c315c7aac"} Jul 31 19:59:28.407493 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:28.407404 2556 generic.go:356] "Generic (PLEG): container finished" podID="c26f3a50-49d3-47df-bd54-bfb9d76361fe" containerID="142a651b4268f69a42674d99c4431ba2871b6d4c5a1297172dd7342c315c7aac" exitCode=0 Jul 31 19:59:28.407493 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:28.407440 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" event={"ID":"c26f3a50-49d3-47df-bd54-bfb9d76361fe","Type":"ContainerDied","Data":"142a651b4268f69a42674d99c4431ba2871b6d4c5a1297172dd7342c315c7aac"} Jul 31 19:59:29.413051 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:29.413016 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" event={"ID":"c26f3a50-49d3-47df-bd54-bfb9d76361fe","Type":"ContainerStarted","Data":"028f79b480245db16bae536bbb2a4355c4c97bea382b8b2e0a805cf0c8fd2852"} Jul 31 19:59:29.413440 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:29.413163 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:29.437093 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:29.437043 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" podStartSLOduration=2.109398064 podStartE2EDuration="6.437029535s" totalImagesPullingTime="3.246194798s" totalInitContainerRuntime="1.081436673s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:29.434801561 +0000 UTC m=+298.776385199" watchObservedRunningTime="2026-07-31 19:59:29.437029535 +0000 UTC m=+298.778613160" Jul 31 19:59:31.067042 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:31.066996 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:59:31.073687 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:31.073655 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 19:59:31.080865 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:31.080841 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 19:59:40.419669 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:40.419638 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-675f995665-8s4gc" Jul 31 19:59:42.652508 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.652474 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f45495bd6-4644m"] Jul 31 19:59:42.655498 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.655462 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.657762 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.657737 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 19:59:42.657891 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.657775 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xrprd\"" Jul 31 19:59:42.658599 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.658577 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 19:59:42.658719 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.658627 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 19:59:42.658778 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.658724 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 19:59:42.666566 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.666542 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f45495bd6-4644m"] Jul 31 19:59:42.727561 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.727539 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7xl\" (UniqueName: \"kubernetes.io/projected/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-kube-api-access-zt7xl\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.727736 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.727572 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tmp\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.727736 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.727599 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tls-certs\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.828448 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.828420 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7xl\" (UniqueName: \"kubernetes.io/projected/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-kube-api-access-zt7xl\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.828574 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.828458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tmp\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.828574 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.828484 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tls-certs\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.830664 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.830637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tmp\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.830842 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.830827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-tls-certs\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.837493 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.837469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7xl\" (UniqueName: \"kubernetes.io/projected/91601de3-a95e-4f35-81b6-a51c2b6d0b1a-kube-api-access-zt7xl\") pod \"kube-auth-proxy-7f45495bd6-4644m\" (UID: \"91601de3-a95e-4f35-81b6-a51c2b6d0b1a\") " pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:42.964719 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:42.964646 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" Jul 31 19:59:43.096125 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.096101 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f45495bd6-4644m"] Jul 31 19:59:43.098324 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:43.098290 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91601de3_a95e_4f35_81b6_a51c2b6d0b1a.slice/crio-c1940743c8215ede74570d0f3468ad6c1fdd2a578485ef325b93244d42cad9ae WatchSource:0}: Error finding container c1940743c8215ede74570d0f3468ad6c1fdd2a578485ef325b93244d42cad9ae: Status 404 returned error can't find the container with id c1940743c8215ede74570d0f3468ad6c1fdd2a578485ef325b93244d42cad9ae Jul 31 19:59:43.100153 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.100137 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:59:43.462424 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.462391 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" event={"ID":"91601de3-a95e-4f35-81b6-a51c2b6d0b1a","Type":"ContainerStarted","Data":"c1940743c8215ede74570d0f3468ad6c1fdd2a578485ef325b93244d42cad9ae"} Jul 31 19:59:43.676205 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.676169 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8kh82"] Jul 31 19:59:43.680401 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.680384 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.683958 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.683943 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 31 19:59:43.684010 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.683964 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-c4hzh\"" Jul 31 19:59:43.700652 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.700629 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8kh82"] Jul 31 19:59:43.736832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.736767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5bx\" (UniqueName: \"kubernetes.io/projected/70fc3499-3444-49f7-a1c9-f0b83e098aae-kube-api-access-ph5bx\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.736832 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.736807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc3499-3444-49f7-a1c9-f0b83e098aae-tls-certs\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.838096 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.838065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc3499-3444-49f7-a1c9-f0b83e098aae-tls-certs\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.838237 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.838216 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5bx\" (UniqueName: \"kubernetes.io/projected/70fc3499-3444-49f7-a1c9-f0b83e098aae-kube-api-access-ph5bx\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.840637 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.840614 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc3499-3444-49f7-a1c9-f0b83e098aae-tls-certs\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.846860 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.846835 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5bx\" (UniqueName: \"kubernetes.io/projected/70fc3499-3444-49f7-a1c9-f0b83e098aae-kube-api-access-ph5bx\") pod \"model-serving-api-54d89bd5fc-8kh82\" (UID: \"70fc3499-3444-49f7-a1c9-f0b83e098aae\") " pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:43.888882 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.888853 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:43.891965 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.891949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:43.894338 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.894299 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 19:59:43.894338 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.894332 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 19:59:43.894495 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.894355 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 19:59:43.894495 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.894394 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ns58k\"" Jul 31 19:59:43.894640 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.894624 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 19:59:43.907269 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.907245 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:43.990478 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:43.990400 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:44.039836 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.039801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.039956 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.039871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.039956 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.039901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.040039 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.040013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.040081 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.040052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.040126 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.040106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.040168 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.040151 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnx5q\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.114399 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.114365 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8kh82"] Jul 31 19:59:44.116844 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:44.116816 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fc3499_3444_49f7_a1c9_f0b83e098aae.slice/crio-281a1ec7d7058b4a002ee7bdf32412712d59b09849070641ce67308e52c60ac7 WatchSource:0}: Error finding container 281a1ec7d7058b4a002ee7bdf32412712d59b09849070641ce67308e52c60ac7: Status 404 returned error can't find the container with id 281a1ec7d7058b4a002ee7bdf32412712d59b09849070641ce67308e52c60ac7 Jul 31 19:59:44.140690 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140769 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140769 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnx5q\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140769 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140769 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140768 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140931 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.140931 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.140818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.141509 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.141481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.142822 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.142797 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.143034 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.143016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.143112 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.143050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.143162 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.143119 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.147764 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.147735 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.147996 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.147978 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnx5q\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q\") pod \"istiod-openshift-gateway-7b7995d5cd-bfrsb\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.201117 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.201089 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:44.344749 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.344587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:44.347335 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:44.347292 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623ee550_f300_46d5_9a75_a37ee085b9ba.slice/crio-9c085dd54e0aaf62dee1f0632c7c8a95c9c586f558bee9b44ced38c5391e1cca WatchSource:0}: Error finding container 9c085dd54e0aaf62dee1f0632c7c8a95c9c586f558bee9b44ced38c5391e1cca: Status 404 returned error can't find the container with id 9c085dd54e0aaf62dee1f0632c7c8a95c9c586f558bee9b44ced38c5391e1cca Jul 31 19:59:44.469635 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.469593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" event={"ID":"70fc3499-3444-49f7-a1c9-f0b83e098aae","Type":"ContainerStarted","Data":"281a1ec7d7058b4a002ee7bdf32412712d59b09849070641ce67308e52c60ac7"} Jul 31 19:59:44.470910 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:44.470880 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" event={"ID":"623ee550-f300-46d5-9a75-a37ee085b9ba","Type":"ContainerStarted","Data":"9c085dd54e0aaf62dee1f0632c7c8a95c9c586f558bee9b44ced38c5391e1cca"} Jul 31 19:59:47.305026 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.304985 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 19:59:47.305427 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.305067 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 31 19:59:47.490010 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.489967 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" event={"ID":"623ee550-f300-46d5-9a75-a37ee085b9ba","Type":"ContainerStarted","Data":"af0981ed627e132d2770a43edb32ccd2d90fea66467e8b8987f143bdbba08309"} Jul 31 19:59:47.490265 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.490128 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:47.491414 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.491389 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" event={"ID":"70fc3499-3444-49f7-a1c9-f0b83e098aae","Type":"ContainerStarted","Data":"3256cfec9ab1a8c57239b3daf6f449354773471e6e031f14204a16c5c8119683"} Jul 31 19:59:47.491538 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.491431 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:47.492808 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.492786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" event={"ID":"91601de3-a95e-4f35-81b6-a51c2b6d0b1a","Type":"ContainerStarted","Data":"16475767f71c3a33af73079c7c1b8f8776719d452c7c7e5348f8f10a8026fed2"} Jul 31 19:59:47.509394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.509348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" podStartSLOduration=1.554148149 podStartE2EDuration="4.50930874s" totalImagesPullingTime="2.955160591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:47.507970571 +0000 UTC m=+316.849554209" watchObservedRunningTime="2026-07-31 19:59:47.50930874 +0000 UTC m=+316.850892366" Jul 31 19:59:47.523370 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.523333 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" podStartSLOduration=1.342793598 podStartE2EDuration="4.523308717s" totalImagesPullingTime="3.180515119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:47.522690528 +0000 UTC m=+316.864274157" watchObservedRunningTime="2026-07-31 19:59:47.523308717 +0000 UTC m=+316.864892342" Jul 31 19:59:47.538673 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:47.538636 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f45495bd6-4644m" podStartSLOduration=1.346584133 podStartE2EDuration="5.538623106s" totalImagesPullingTime="4.192038973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:47.537226265 +0000 UTC m=+316.878809891" watchObservedRunningTime="2026-07-31 19:59:47.538623106 +0000 UTC m=+316.880206731" Jul 31 19:59:48.501982 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:48.501946 2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bfrsb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 19:59:48.502344 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:48.502008 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 19:59:51.500894 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:51.500865 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:53.442804 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.442769 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd"] Jul 31 19:59:53.445889 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.445867 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.448792 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.448775 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sjdm6\"" Jul 31 19:59:53.449307 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.449290 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 31 19:59:53.470766 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.470735 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd"] Jul 31 19:59:53.472113 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.472090 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv"] Jul 31 19:59:53.475197 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.475177 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:53.477327 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.477300 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-q5hc6\"" Jul 31 19:59:53.477491 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.477479 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 31 19:59:53.490452 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.490434 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv"] Jul 31 19:59:53.522743 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.522716 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkvx\" (UniqueName: \"kubernetes.io/projected/1922c780-f7c8-4e28-81ed-27375ff15224-kube-api-access-4mkvx\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.522836 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.522745 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.623863 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.623833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:53.624002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.623872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkvx\" (UniqueName: \"kubernetes.io/projected/1922c780-f7c8-4e28-81ed-27375ff15224-kube-api-access-4mkvx\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.624002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.623890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.624002 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.623962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtnv\" (UniqueName: \"kubernetes.io/projected/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-kube-api-access-hjtnv\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:53.624002 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:53.623969 2556 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 31 19:59:53.624146 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:53.624089 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert podName:1922c780-f7c8-4e28-81ed-27375ff15224 nodeName:}" failed. No retries permitted until 2026-07-31 19:59:54.124067272 +0000 UTC m=+323.465650901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert") pod "kserve-controller-manager-d6f4b65b8-zj2jd" (UID: "1922c780-f7c8-4e28-81ed-27375ff15224") : secret "kserve-webhook-server-cert" not found Jul 31 19:59:53.642830 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.642807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkvx\" (UniqueName: \"kubernetes.io/projected/1922c780-f7c8-4e28-81ed-27375ff15224-kube-api-access-4mkvx\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:53.725030 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.724968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:53.725030 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.725032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtnv\" (UniqueName: \"kubernetes.io/projected/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-kube-api-access-hjtnv\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:53.725152 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:53.725095 2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 31 19:59:53.725152 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:53.725144 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert podName:b74953c6-c6d3-47ff-83a5-8fbdac3d2c96 nodeName:}" failed. No retries permitted until 2026-07-31 19:59:54.225132653 +0000 UTC m=+323.566716257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert") pod "llmisvc-controller-manager-777c6bd7d5-79mbv" (UID: "b74953c6-c6d3-47ff-83a5-8fbdac3d2c96") : secret "llmisvc-webhook-server-cert" not found Jul 31 19:59:53.735361 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:53.735337 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtnv\" (UniqueName: \"kubernetes.io/projected/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-kube-api-access-hjtnv\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:54.128188 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.128140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:54.128387 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:54.128289 2556 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 31 19:59:54.128387 ip-10-0-143-178 kubenswrapper[2556]: E0731 19:59:54.128378 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert podName:1922c780-f7c8-4e28-81ed-27375ff15224 nodeName:}" failed. No retries permitted until 2026-07-31 19:59:55.128363503 +0000 UTC m=+324.469947110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert") pod "kserve-controller-manager-d6f4b65b8-zj2jd" (UID: "1922c780-f7c8-4e28-81ed-27375ff15224") : secret "kserve-webhook-server-cert" not found Jul 31 19:59:54.228963 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.228936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:54.231320 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.231293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74953c6-c6d3-47ff-83a5-8fbdac3d2c96-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-79mbv\" (UID: \"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:54.385890 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.385817 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:54.507506 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.507483 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv"] Jul 31 19:59:54.509298 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:54.509269 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb74953c6_c6d3_47ff_83a5_8fbdac3d2c96.slice/crio-6f1c96c631b65a5dbff6df34a8423a2c73d9fd2588a8464e1d56efdaba9406cb WatchSource:0}: Error finding container 6f1c96c631b65a5dbff6df34a8423a2c73d9fd2588a8464e1d56efdaba9406cb: Status 404 returned error can't find the container with id 6f1c96c631b65a5dbff6df34a8423a2c73d9fd2588a8464e1d56efdaba9406cb Jul 31 19:59:54.525644 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:54.525619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" event={"ID":"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96","Type":"ContainerStarted","Data":"6f1c96c631b65a5dbff6df34a8423a2c73d9fd2588a8464e1d56efdaba9406cb"} Jul 31 19:59:55.138444 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:55.138403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:55.140499 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:55.140474 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1922c780-f7c8-4e28-81ed-27375ff15224-cert\") pod \"kserve-controller-manager-d6f4b65b8-zj2jd\" (UID: \"1922c780-f7c8-4e28-81ed-27375ff15224\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:55.255485 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:55.255460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:55.376576 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:55.376556 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd"] Jul 31 19:59:55.378980 ip-10-0-143-178 kubenswrapper[2556]: W0731 19:59:55.378953 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1922c780_f7c8_4e28_81ed_27375ff15224.slice/crio-ffedad84e908e966a17bf2a9d8cd0de1b9eed8ef2befc6c382d9d7c85b4620c1 WatchSource:0}: Error finding container ffedad84e908e966a17bf2a9d8cd0de1b9eed8ef2befc6c382d9d7c85b4620c1: Status 404 returned error can't find the container with id ffedad84e908e966a17bf2a9d8cd0de1b9eed8ef2befc6c382d9d7c85b4620c1 Jul 31 19:59:55.532147 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:55.532060 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" event={"ID":"1922c780-f7c8-4e28-81ed-27375ff15224","Type":"ContainerStarted","Data":"ffedad84e908e966a17bf2a9d8cd0de1b9eed8ef2befc6c382d9d7c85b4620c1"} Jul 31 19:59:56.557282 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:56.557245 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:56.557752 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:56.557493 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" containerID="cri-o://af0981ed627e132d2770a43edb32ccd2d90fea66467e8b8987f143bdbba08309" gracePeriod=30 Jul 31 19:59:57.544273 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.544178 2556 generic.go:356] "Generic (PLEG): container finished" podID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerID="af0981ed627e132d2770a43edb32ccd2d90fea66467e8b8987f143bdbba08309" exitCode=0 Jul 31 19:59:57.544273 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.544224 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" event={"ID":"623ee550-f300-46d5-9a75-a37ee085b9ba","Type":"ContainerDied","Data":"af0981ed627e132d2770a43edb32ccd2d90fea66467e8b8987f143bdbba08309"} Jul 31 19:59:57.715364 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.715338 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:57.860605 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860571 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860732 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860624 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860732 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860680 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860732 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860723 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860903 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860747 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnx5q\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860903 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860796 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.860903 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.860854 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs\") pod \"623ee550-f300-46d5-9a75-a37ee085b9ba\" (UID: \"623ee550-f300-46d5-9a75-a37ee085b9ba\") " Jul 31 19:59:57.861362 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.861249 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:59:57.863141 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863106 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts" (OuterVolumeSpecName: "cacerts") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:59:57.863224 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863158 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:59:57.863224 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863184 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:59:57.863458 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863396 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q" (OuterVolumeSpecName: "kube-api-access-cnx5q") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "kube-api-access-cnx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:59:57.863603 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863585 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:59:57.863976 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.863953 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs" (OuterVolumeSpecName: "local-certs") pod "623ee550-f300-46d5-9a75-a37ee085b9ba" (UID: "623ee550-f300-46d5-9a75-a37ee085b9ba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:59:57.961507 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961475 2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-dns-cert\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961507 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961505 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-token\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961507 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961515 2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-kubeconfig\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961727 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961523 2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/623ee550-f300-46d5-9a75-a37ee085b9ba-cacerts\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961727 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961533 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnx5q\" (UniqueName: \"kubernetes.io/projected/623ee550-f300-46d5-9a75-a37ee085b9ba-kube-api-access-cnx5q\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961727 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961542 2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/623ee550-f300-46d5-9a75-a37ee085b9ba-istio-csr-ca-configmap\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:57.961727 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:57.961552 2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee550-f300-46d5-9a75-a37ee085b9ba-local-certs\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 19:59:58.504000 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.503970 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8kh82" Jul 31 19:59:58.550692 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.550663 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" event={"ID":"1922c780-f7c8-4e28-81ed-27375ff15224","Type":"ContainerStarted","Data":"3f85740f2f8f9346499bc88b14c836c612b5c92e162eb878e655e8ae5927667d"} Jul 31 19:59:58.550846 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.550735 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 19:59:58.552237 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.552210 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" event={"ID":"b74953c6-c6d3-47ff-83a5-8fbdac3d2c96","Type":"ContainerStarted","Data":"f6b783fc9378d1eca2def116c4fc21b464225631650b25ce429105307b9ff880"} Jul 31 19:59:58.552394 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.552273 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 19:59:58.553444 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.553420 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" event={"ID":"623ee550-f300-46d5-9a75-a37ee085b9ba","Type":"ContainerDied","Data":"9c085dd54e0aaf62dee1f0632c7c8a95c9c586f558bee9b44ced38c5391e1cca"} Jul 31 19:59:58.553554 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.553454 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" Jul 31 19:59:58.553554 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.553469 2556 scope.go:117] "RemoveContainer" containerID="af0981ed627e132d2770a43edb32ccd2d90fea66467e8b8987f143bdbba08309" Jul 31 19:59:58.566234 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.566188 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" podStartSLOduration=3.213793418 podStartE2EDuration="5.566174076s" totalImagesPullingTime="2.352380658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:58.565606595 +0000 UTC m=+327.907190223" watchObservedRunningTime="2026-07-31 19:59:58.566174076 +0000 UTC m=+327.907757703" Jul 31 19:59:58.580634 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.580611 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:58.586609 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.586590 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb"] Jul 31 19:59:58.600476 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:58.600433 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" podStartSLOduration=2.379936259 podStartE2EDuration="5.600420578s" totalImagesPullingTime="3.220484319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:59:58.599302832 +0000 UTC m=+327.940886462" watchObservedRunningTime="2026-07-31 19:59:58.600420578 +0000 UTC m=+327.942004204" Jul 31 19:59:59.167823 ip-10-0-143-178 kubenswrapper[2556]: I0731 19:59:59.167788 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" path="/var/lib/kubelet/pods/623ee550-f300-46d5-9a75-a37ee085b9ba/volumes" Jul 31 20:00:02.500830 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:02.500791 2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bfrsb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 31 20:00:02.501197 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:02.500862 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfrsb" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.20:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 20:00:29.560741 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:29.560659 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-79mbv" Jul 31 20:00:29.563701 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:29.563677 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zj2jd" Jul 31 20:00:46.723339 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.723276 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q"] Jul 31 20:00:46.723905 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.723690 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" Jul 31 20:00:46.723905 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.723708 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" Jul 31 20:00:46.723905 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.723787 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="623ee550-f300-46d5-9a75-a37ee085b9ba" containerName="discovery" Jul 31 20:00:46.730658 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.730635 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.733589 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.733566 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 20:00:46.733818 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.733787 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q"] Jul 31 20:00:46.734502 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.734480 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 20:00:46.734602 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.734522 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nbqwr\"" Jul 31 20:00:46.793696 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.793659 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.793696 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.793703 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.793939 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.793784 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z69h\" (UniqueName: \"kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.825135 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.825102 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z"] Jul 31 20:00:46.828347 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.828332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.835369 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.835344 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z"] Jul 31 20:00:46.894437 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.894608 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.894608 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74d2\" (UniqueName: \"kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.894608 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.894608 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.894760 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z69h\" (UniqueName: \"kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.894894 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.894938 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.894886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.901994 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.901969 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z69h\" (UniqueName: \"kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:46.922066 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.922040 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4"] Jul 31 20:00:46.925429 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.925413 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:46.932912 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.932890 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4"] Jul 31 20:00:46.995966 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.995870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.995966 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.995927 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:46.996190 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996001 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:46.996190 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.996190 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996082 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74d2\" (UniqueName: \"kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.996190 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qv4\" (UniqueName: \"kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:46.996390 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996367 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:46.996431 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:46.996380 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:47.005845 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.005824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74d2\" (UniqueName: \"kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:47.024403 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.024376 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs"] Jul 31 20:00:47.027778 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.027763 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.035843 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.035822 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs"] Jul 31 20:00:47.041857 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.041839 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:47.096935 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.096894 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.097040 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.096969 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.097082 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2w5t\" (UniqueName: \"kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.097116 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097098 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.097180 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.097248 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.097284 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qv4\" (UniqueName: \"kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.097477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.097457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.106407 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.106374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qv4\" (UniqueName: \"kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.137418 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.137381 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:47.175102 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.175078 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q"] Jul 31 20:00:47.177539 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:00:47.177512 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4014386e_e9be_47b5_8709_6c1eb72a4ed6.slice/crio-60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f WatchSource:0}: Error finding container 60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f: Status 404 returned error can't find the container with id 60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f Jul 31 20:00:47.198133 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.198103 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.198245 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.198157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2w5t\" (UniqueName: \"kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.198245 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.198187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.198671 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.198590 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.198671 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.198627 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.206464 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.206441 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2w5t\" (UniqueName: \"kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.234729 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.234701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:47.275339 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.275292 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z"] Jul 31 20:00:47.276306 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:00:47.276277 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61eee60d_2958_4d82_af5a_c8e29f6237fc.slice/crio-5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5 WatchSource:0}: Error finding container 5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5: Status 404 returned error can't find the container with id 5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5 Jul 31 20:00:47.337805 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.337768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:47.370982 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.370951 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4"] Jul 31 20:00:47.477783 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.477759 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs"] Jul 31 20:00:47.553201 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:00:47.553128 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5d8139_567e_4207_a605_05b53e147e88.slice/crio-7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be WatchSource:0}: Error finding container 7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be: Status 404 returned error can't find the container with id 7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be Jul 31 20:00:47.752986 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.752950 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e5d8139-567e-4207-a605-05b53e147e88" containerID="801e956188a7945f0a99fb11d7153bd7a2dc1ce5ac294e67adf3823369c3ea4d" exitCode=0 Jul 31 20:00:47.753435 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.753036 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerDied","Data":"801e956188a7945f0a99fb11d7153bd7a2dc1ce5ac294e67adf3823369c3ea4d"} Jul 31 20:00:47.753435 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.753087 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerStarted","Data":"7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be"} Jul 31 20:00:47.754587 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.754565 2556 generic.go:356] "Generic (PLEG): container finished" podID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerID="ebe46674791c1038277e3913eb6ae79a28d3b2d7d097324090afb58bf8480906" exitCode=0 Jul 31 20:00:47.754672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.754654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" event={"ID":"61eee60d-2958-4d82-af5a-c8e29f6237fc","Type":"ContainerDied","Data":"ebe46674791c1038277e3913eb6ae79a28d3b2d7d097324090afb58bf8480906"} Jul 31 20:00:47.754735 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.754689 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" event={"ID":"61eee60d-2958-4d82-af5a-c8e29f6237fc","Type":"ContainerStarted","Data":"5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5"} Jul 31 20:00:47.756179 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.756159 2556 generic.go:356] "Generic (PLEG): container finished" podID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerID="dbb6212838cdf0dfa5748a8a1c39f0a56abfced3b33afa3924e47be36591b7b4" exitCode=0 Jul 31 20:00:47.756270 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.756229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" event={"ID":"4014386e-e9be-47b5-8709-6c1eb72a4ed6","Type":"ContainerDied","Data":"dbb6212838cdf0dfa5748a8a1c39f0a56abfced3b33afa3924e47be36591b7b4"} Jul 31 20:00:47.756355 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.756273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" event={"ID":"4014386e-e9be-47b5-8709-6c1eb72a4ed6","Type":"ContainerStarted","Data":"60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f"} Jul 31 20:00:47.757879 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.757858 2556 generic.go:356] "Generic (PLEG): container finished" podID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerID="363a41a9ab51ee895f5fd3829d579f927f90ff80dd9f21169db1dae142c95579" exitCode=0 Jul 31 20:00:47.757961 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.757913 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" event={"ID":"af12d7b1-c0fe-4683-b23f-39c4312d0943","Type":"ContainerDied","Data":"363a41a9ab51ee895f5fd3829d579f927f90ff80dd9f21169db1dae142c95579"} Jul 31 20:00:47.757961 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:47.757942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" event={"ID":"af12d7b1-c0fe-4683-b23f-39c4312d0943","Type":"ContainerStarted","Data":"c69a3c548c155bb0c4a7c89ef5b3e6c5122383541fd84c7ba62f6d7c5ee8db87"} Jul 31 20:00:48.768237 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:48.768202 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerStarted","Data":"a6293ffe70d08c1953d9ea569ab0ed972e9ef95cdd58f880c5bb11a07bbc7d1b"} Jul 31 20:00:48.770018 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:48.769994 2556 generic.go:356] "Generic (PLEG): container finished" podID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerID="c79ba00e0d1b894504d5332dc1589fca9932672ba9e87fff7f336a6156198700" exitCode=0 Jul 31 20:00:48.770115 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:48.770051 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" event={"ID":"af12d7b1-c0fe-4683-b23f-39c4312d0943","Type":"ContainerDied","Data":"c79ba00e0d1b894504d5332dc1589fca9932672ba9e87fff7f336a6156198700"} Jul 31 20:00:49.776727 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.776648 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e5d8139-567e-4207-a605-05b53e147e88" containerID="a6293ffe70d08c1953d9ea569ab0ed972e9ef95cdd58f880c5bb11a07bbc7d1b" exitCode=0 Jul 31 20:00:49.777157 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.776735 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerDied","Data":"a6293ffe70d08c1953d9ea569ab0ed972e9ef95cdd58f880c5bb11a07bbc7d1b"} Jul 31 20:00:49.778618 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.778589 2556 generic.go:356] "Generic (PLEG): container finished" podID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerID="4301a083b5d899a12e95c759cc3078fb2c292e7222ab6a832df370e9efc682ce" exitCode=0 Jul 31 20:00:49.778720 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.778641 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" event={"ID":"61eee60d-2958-4d82-af5a-c8e29f6237fc","Type":"ContainerDied","Data":"4301a083b5d899a12e95c759cc3078fb2c292e7222ab6a832df370e9efc682ce"} Jul 31 20:00:49.780965 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.780940 2556 generic.go:356] "Generic (PLEG): container finished" podID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerID="aefbda60bf7f8a64588516221238bc229a85108b2561f1e5f01c20dbed520921" exitCode=0 Jul 31 20:00:49.781043 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:49.780989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" event={"ID":"af12d7b1-c0fe-4683-b23f-39c4312d0943","Type":"ContainerDied","Data":"aefbda60bf7f8a64588516221238bc229a85108b2561f1e5f01c20dbed520921"} Jul 31 20:00:50.787934 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.787896 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e5d8139-567e-4207-a605-05b53e147e88" containerID="95486bbf99072d3f8275925dbef9dd1c4ecf476c159b73c8a90b4989ac91edda" exitCode=0 Jul 31 20:00:50.788415 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.787975 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerDied","Data":"95486bbf99072d3f8275925dbef9dd1c4ecf476c159b73c8a90b4989ac91edda"} Jul 31 20:00:50.790024 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.790002 2556 generic.go:356] "Generic (PLEG): container finished" podID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerID="6f2e9f3930674d29c581185f0eabb4ccd999fee5ad60c626f8e20228a6abec07" exitCode=0 Jul 31 20:00:50.790122 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.790082 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" event={"ID":"61eee60d-2958-4d82-af5a-c8e29f6237fc","Type":"ContainerDied","Data":"6f2e9f3930674d29c581185f0eabb4ccd999fee5ad60c626f8e20228a6abec07"} Jul 31 20:00:50.791587 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.791568 2556 generic.go:356] "Generic (PLEG): container finished" podID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerID="0ea5f9abd4af920d3f89c86afd9a3891cdc286fa8a22ef95d3147ec299d35e3c" exitCode=0 Jul 31 20:00:50.791672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.791652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" event={"ID":"4014386e-e9be-47b5-8709-6c1eb72a4ed6","Type":"ContainerDied","Data":"0ea5f9abd4af920d3f89c86afd9a3891cdc286fa8a22ef95d3147ec299d35e3c"} Jul 31 20:00:50.924065 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:50.924036 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:51.031191 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.031157 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qv4\" (UniqueName: \"kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4\") pod \"af12d7b1-c0fe-4683-b23f-39c4312d0943\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " Jul 31 20:00:51.031191 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.031202 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle\") pod \"af12d7b1-c0fe-4683-b23f-39c4312d0943\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " Jul 31 20:00:51.031422 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.031242 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util\") pod \"af12d7b1-c0fe-4683-b23f-39c4312d0943\" (UID: \"af12d7b1-c0fe-4683-b23f-39c4312d0943\") " Jul 31 20:00:51.031731 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.031702 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle" (OuterVolumeSpecName: "bundle") pod "af12d7b1-c0fe-4683-b23f-39c4312d0943" (UID: "af12d7b1-c0fe-4683-b23f-39c4312d0943"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:51.033268 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.033244 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4" (OuterVolumeSpecName: "kube-api-access-75qv4") pod "af12d7b1-c0fe-4683-b23f-39c4312d0943" (UID: "af12d7b1-c0fe-4683-b23f-39c4312d0943"). InnerVolumeSpecName "kube-api-access-75qv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:00:51.038234 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.038174 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util" (OuterVolumeSpecName: "util") pod "af12d7b1-c0fe-4683-b23f-39c4312d0943" (UID: "af12d7b1-c0fe-4683-b23f-39c4312d0943"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:51.131816 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.131782 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:51.131816 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.131812 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qv4\" (UniqueName: \"kubernetes.io/projected/af12d7b1-c0fe-4683-b23f-39c4312d0943-kube-api-access-75qv4\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:51.131816 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.131823 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af12d7b1-c0fe-4683-b23f-39c4312d0943-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:51.798526 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.798493 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" Jul 31 20:00:51.798962 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.798519 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb5gld4" event={"ID":"af12d7b1-c0fe-4683-b23f-39c4312d0943","Type":"ContainerDied","Data":"c69a3c548c155bb0c4a7c89ef5b3e6c5122383541fd84c7ba62f6d7c5ee8db87"} Jul 31 20:00:51.798962 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.798609 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69a3c548c155bb0c4a7c89ef5b3e6c5122383541fd84c7ba62f6d7c5ee8db87" Jul 31 20:00:51.800843 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.800813 2556 generic.go:356] "Generic (PLEG): container finished" podID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerID="be0010f50314cf90c57f495cb7269c532cc7500f71269f8e2b9e8eeb22412cef" exitCode=0 Jul 31 20:00:51.800984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.800853 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" event={"ID":"4014386e-e9be-47b5-8709-6c1eb72a4ed6","Type":"ContainerDied","Data":"be0010f50314cf90c57f495cb7269c532cc7500f71269f8e2b9e8eeb22412cef"} Jul 31 20:00:51.928586 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.928558 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:51.958597 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:51.958573 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:52.038484 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038451 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util\") pod \"4e5d8139-567e-4207-a605-05b53e147e88\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " Jul 31 20:00:52.038484 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038493 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle\") pod \"61eee60d-2958-4d82-af5a-c8e29f6237fc\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " Jul 31 20:00:52.038782 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038514 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v74d2\" (UniqueName: \"kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2\") pod \"61eee60d-2958-4d82-af5a-c8e29f6237fc\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " Jul 31 20:00:52.038782 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038534 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle\") pod \"4e5d8139-567e-4207-a605-05b53e147e88\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " Jul 31 20:00:52.038782 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038558 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util\") pod \"61eee60d-2958-4d82-af5a-c8e29f6237fc\" (UID: \"61eee60d-2958-4d82-af5a-c8e29f6237fc\") " Jul 31 20:00:52.038782 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.038593 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2w5t\" (UniqueName: \"kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t\") pod \"4e5d8139-567e-4207-a605-05b53e147e88\" (UID: \"4e5d8139-567e-4207-a605-05b53e147e88\") " Jul 31 20:00:52.039063 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.039037 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle" (OuterVolumeSpecName: "bundle") pod "4e5d8139-567e-4207-a605-05b53e147e88" (UID: "4e5d8139-567e-4207-a605-05b53e147e88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:52.039157 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.039115 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle" (OuterVolumeSpecName: "bundle") pod "61eee60d-2958-4d82-af5a-c8e29f6237fc" (UID: "61eee60d-2958-4d82-af5a-c8e29f6237fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:52.040661 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.040632 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2" (OuterVolumeSpecName: "kube-api-access-v74d2") pod "61eee60d-2958-4d82-af5a-c8e29f6237fc" (UID: "61eee60d-2958-4d82-af5a-c8e29f6237fc"). InnerVolumeSpecName "kube-api-access-v74d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:00:52.040793 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.040673 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t" (OuterVolumeSpecName: "kube-api-access-v2w5t") pod "4e5d8139-567e-4207-a605-05b53e147e88" (UID: "4e5d8139-567e-4207-a605-05b53e147e88"). InnerVolumeSpecName "kube-api-access-v2w5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:00:52.044534 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.044508 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util" (OuterVolumeSpecName: "util") pod "61eee60d-2958-4d82-af5a-c8e29f6237fc" (UID: "61eee60d-2958-4d82-af5a-c8e29f6237fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:52.047011 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.046989 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util" (OuterVolumeSpecName: "util") pod "4e5d8139-567e-4207-a605-05b53e147e88" (UID: "4e5d8139-567e-4207-a605-05b53e147e88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:52.139091 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139060 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.139091 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139087 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.139091 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139096 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74d2\" (UniqueName: \"kubernetes.io/projected/61eee60d-2958-4d82-af5a-c8e29f6237fc-kube-api-access-v74d2\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.139329 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139130 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8139-567e-4207-a605-05b53e147e88-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.139329 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139139 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61eee60d-2958-4d82-af5a-c8e29f6237fc-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.139329 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.139147 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2w5t\" (UniqueName: \"kubernetes.io/projected/4e5d8139-567e-4207-a605-05b53e147e88-kube-api-access-v2w5t\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:52.807494 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.807466 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" Jul 31 20:00:52.807494 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.807472 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08z5qs" event={"ID":"4e5d8139-567e-4207-a605-05b53e147e88","Type":"ContainerDied","Data":"7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be"} Jul 31 20:00:52.808033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.807528 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7124e6ea781fe3888301b63406c52e7bbf1c02f22799c9f3785a4895de9186be" Jul 31 20:00:52.809388 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.809357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" event={"ID":"61eee60d-2958-4d82-af5a-c8e29f6237fc","Type":"ContainerDied","Data":"5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5"} Jul 31 20:00:52.809524 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.809401 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crd75z" Jul 31 20:00:52.809524 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.809405 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b4a6024a7dbfcb12183b1d1c313da08b85a9df2b0b24dfd680ec7b08fd8c4c5" Jul 31 20:00:52.937637 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:52.937612 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:53.045974 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.045938 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z69h\" (UniqueName: \"kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h\") pod \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " Jul 31 20:00:53.046114 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.046002 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle\") pod \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " Jul 31 20:00:53.046114 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.046030 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util\") pod \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\" (UID: \"4014386e-e9be-47b5-8709-6c1eb72a4ed6\") " Jul 31 20:00:53.046684 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.046659 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle" (OuterVolumeSpecName: "bundle") pod "4014386e-e9be-47b5-8709-6c1eb72a4ed6" (UID: "4014386e-e9be-47b5-8709-6c1eb72a4ed6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:53.047996 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.047970 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h" (OuterVolumeSpecName: "kube-api-access-6z69h") pod "4014386e-e9be-47b5-8709-6c1eb72a4ed6" (UID: "4014386e-e9be-47b5-8709-6c1eb72a4ed6"). InnerVolumeSpecName "kube-api-access-6z69h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:00:53.050978 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.050945 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util" (OuterVolumeSpecName: "util") pod "4014386e-e9be-47b5-8709-6c1eb72a4ed6" (UID: "4014386e-e9be-47b5-8709-6c1eb72a4ed6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:00:53.147151 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.147064 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-bundle\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:53.147151 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.147094 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4014386e-e9be-47b5-8709-6c1eb72a4ed6-util\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:53.147151 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.147103 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z69h\" (UniqueName: \"kubernetes.io/projected/4014386e-e9be-47b5-8709-6c1eb72a4ed6-kube-api-access-6z69h\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:00:53.816078 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.816048 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" Jul 31 20:00:53.816575 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.816049 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vmj7q" event={"ID":"4014386e-e9be-47b5-8709-6c1eb72a4ed6","Type":"ContainerDied","Data":"60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f"} Jul 31 20:00:53.816575 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:00:53.816197 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f1ffeacc3070429f791ca9680c44d2f2056b0d4fad06149c1801d9d381b00f" Jul 31 20:01:04.792138 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792100 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh"] Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792382 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792393 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792403 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792409 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792417 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792422 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792430 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792436 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792442 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792447 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792454 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792459 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792466 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792471 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792477 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792482 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="extract" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792489 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792494 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792502 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792507 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792513 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792518 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="util" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792522 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="pull" Jul 31 20:01:04.792533 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792528 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="pull" Jul 31 20:01:04.793178 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792573 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4014386e-e9be-47b5-8709-6c1eb72a4ed6" containerName="extract" Jul 31 20:01:04.793178 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792582 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5d8139-567e-4207-a605-05b53e147e88" containerName="extract" Jul 31 20:01:04.793178 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792587 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="61eee60d-2958-4d82-af5a-c8e29f6237fc" containerName="extract" Jul 31 20:01:04.793178 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.792593 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af12d7b1-c0fe-4683-b23f-39c4312d0943" containerName="extract" Jul 31 20:01:04.795593 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.795573 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:04.797848 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.797818 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 31 20:01:04.797968 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.797822 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mj5vx\"" Jul 31 20:01:04.797968 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.797824 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 20:01:04.797968 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.797824 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 20:01:04.806816 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.806788 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh"] Jul 31 20:01:04.837431 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.837391 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6k6j\" (UniqueName: \"kubernetes.io/projected/dd0c848b-6709-43cb-90b2-787ae5985fab-kube-api-access-g6k6j\") pod \"dns-operator-controller-manager-5985564776-d8xgh\" (UID: \"dd0c848b-6709-43cb-90b2-787ae5985fab\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:04.938688 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.938625 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6k6j\" (UniqueName: \"kubernetes.io/projected/dd0c848b-6709-43cb-90b2-787ae5985fab-kube-api-access-g6k6j\") pod \"dns-operator-controller-manager-5985564776-d8xgh\" (UID: \"dd0c848b-6709-43cb-90b2-787ae5985fab\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:04.951003 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:04.950967 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6k6j\" (UniqueName: \"kubernetes.io/projected/dd0c848b-6709-43cb-90b2-787ae5985fab-kube-api-access-g6k6j\") pod \"dns-operator-controller-manager-5985564776-d8xgh\" (UID: \"dd0c848b-6709-43cb-90b2-787ae5985fab\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:05.105512 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:05.105477 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:05.241530 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:05.241499 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh"] Jul 31 20:01:05.243751 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:05.243719 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c848b_6709_43cb_90b2_787ae5985fab.slice/crio-be664277dcd98a6f79ec9572c6d7c2ebfedc28a346f5f44e884e860ba22eb893 WatchSource:0}: Error finding container be664277dcd98a6f79ec9572c6d7c2ebfedc28a346f5f44e884e860ba22eb893: Status 404 returned error can't find the container with id be664277dcd98a6f79ec9572c6d7c2ebfedc28a346f5f44e884e860ba22eb893 Jul 31 20:01:05.877289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:05.877246 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" event={"ID":"dd0c848b-6709-43cb-90b2-787ae5985fab","Type":"ContainerStarted","Data":"be664277dcd98a6f79ec9572c6d7c2ebfedc28a346f5f44e884e860ba22eb893"} Jul 31 20:01:08.882427 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.882391 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5"] Jul 31 20:01:08.885921 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.885899 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:08.888242 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.888220 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j2f22\"" Jul 31 20:01:08.895733 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.895705 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" event={"ID":"dd0c848b-6709-43cb-90b2-787ae5985fab","Type":"ContainerStarted","Data":"054d29749c71e85677c0d6bacbf215e6ae4b796edb39430eab781aba2a68725a"} Jul 31 20:01:08.895844 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.895768 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:08.897845 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.897823 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5"] Jul 31 20:01:08.939432 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.939378 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" podStartSLOduration=2.332950419 podStartE2EDuration="4.939362643s" totalImagesPullingTime="2.606412224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:08.938097462 +0000 UTC m=+398.279681089" watchObservedRunningTime="2026-07-31 20:01:08.939362643 +0000 UTC m=+398.280946271" Jul 31 20:01:08.974014 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.973975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdk5c\" (UniqueName: \"kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:08.974154 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:08.974091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.075192 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.075156 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.075383 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.075207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdk5c\" (UniqueName: \"kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.075647 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.075623 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.083950 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.083929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdk5c\" (UniqueName: \"kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c\") pod \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.197352 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.197251 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:09.347562 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.347536 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5"] Jul 31 20:01:09.350330 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:09.350281 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31485a40_892c_4024_b9bf_08c5e4d7ec78.slice/crio-854c11f1dc683d7b2e954a6b7043ecab95fee5f4b4b655bc0a09184c041da98c WatchSource:0}: Error finding container 854c11f1dc683d7b2e954a6b7043ecab95fee5f4b4b655bc0a09184c041da98c: Status 404 returned error can't find the container with id 854c11f1dc683d7b2e954a6b7043ecab95fee5f4b4b655bc0a09184c041da98c Jul 31 20:01:09.901320 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:09.901274 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" event={"ID":"31485a40-892c-4024-b9bf-08c5e4d7ec78","Type":"ContainerStarted","Data":"854c11f1dc683d7b2e954a6b7043ecab95fee5f4b4b655bc0a09184c041da98c"} Jul 31 20:01:11.531113 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.531077 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qnwrs"] Jul 31 20:01:11.534294 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.534271 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:11.538052 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.538025 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7j5d9\"" Jul 31 20:01:11.545766 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.545745 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qnwrs"] Jul 31 20:01:11.594530 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.594491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8f9\" (UniqueName: \"kubernetes.io/projected/592a1350-8bf6-490b-8509-75502d3b1d31-kube-api-access-qf8f9\") pod \"authorino-operator-68d88948d8-qnwrs\" (UID: \"592a1350-8bf6-490b-8509-75502d3b1d31\") " pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:11.695240 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.695165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8f9\" (UniqueName: \"kubernetes.io/projected/592a1350-8bf6-490b-8509-75502d3b1d31-kube-api-access-qf8f9\") pod \"authorino-operator-68d88948d8-qnwrs\" (UID: \"592a1350-8bf6-490b-8509-75502d3b1d31\") " pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:11.709747 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.709716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8f9\" (UniqueName: \"kubernetes.io/projected/592a1350-8bf6-490b-8509-75502d3b1d31-kube-api-access-qf8f9\") pod \"authorino-operator-68d88948d8-qnwrs\" (UID: \"592a1350-8bf6-490b-8509-75502d3b1d31\") " pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:11.847545 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.847515 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:11.994126 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:11.994102 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qnwrs"] Jul 31 20:01:11.996579 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:11.996549 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592a1350_8bf6_490b_8509_75502d3b1d31.slice/crio-eb4482b80c25e2e1964ac99ff6ee93dc02c0d9b2a5f783373327767cd7bf048e WatchSource:0}: Error finding container eb4482b80c25e2e1964ac99ff6ee93dc02c0d9b2a5f783373327767cd7bf048e: Status 404 returned error can't find the container with id eb4482b80c25e2e1964ac99ff6ee93dc02c0d9b2a5f783373327767cd7bf048e Jul 31 20:01:12.919690 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:12.919642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" event={"ID":"592a1350-8bf6-490b-8509-75502d3b1d31","Type":"ContainerStarted","Data":"eb4482b80c25e2e1964ac99ff6ee93dc02c0d9b2a5f783373327767cd7bf048e"} Jul 31 20:01:13.766998 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:13.766957 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8"] Jul 31 20:01:13.770752 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:13.770722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:13.774093 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:13.774068 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4wg7x\"" Jul 31 20:01:13.783748 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:13.783726 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8"] Jul 31 20:01:13.914494 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:13.914455 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf927\" (UniqueName: \"kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927\") pod \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" (UID: \"7eaae507-624a-42c3-9f2f-46f5a0a66c7a\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:14.014868 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.014830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf927\" (UniqueName: \"kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927\") pod \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" (UID: \"7eaae507-624a-42c3-9f2f-46f5a0a66c7a\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:14.023665 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.023643 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf927\" (UniqueName: \"kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927\") pod \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" (UID: \"7eaae507-624a-42c3-9f2f-46f5a0a66c7a\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:14.082753 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.082729 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:14.232070 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.232046 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8"] Jul 31 20:01:14.234900 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:14.234874 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaae507_624a_42c3_9f2f_46f5a0a66c7a.slice/crio-0105600d86657ca918378b455187f7093da511a9e872b9fcee897d93487dec10 WatchSource:0}: Error finding container 0105600d86657ca918378b455187f7093da511a9e872b9fcee897d93487dec10: Status 404 returned error can't find the container with id 0105600d86657ca918378b455187f7093da511a9e872b9fcee897d93487dec10 Jul 31 20:01:14.930267 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.930229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" event={"ID":"7eaae507-624a-42c3-9f2f-46f5a0a66c7a","Type":"ContainerStarted","Data":"0105600d86657ca918378b455187f7093da511a9e872b9fcee897d93487dec10"} Jul 31 20:01:14.931904 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.931871 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" event={"ID":"31485a40-892c-4024-b9bf-08c5e4d7ec78","Type":"ContainerStarted","Data":"f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896"} Jul 31 20:01:14.932061 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.932044 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:14.933515 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.933487 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" event={"ID":"592a1350-8bf6-490b-8509-75502d3b1d31","Type":"ContainerStarted","Data":"a29a307ded8f681896cb5592db1b3fa5acc0e75c6c07853e6df51b5b31c9dfda"} Jul 31 20:01:14.933601 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.933537 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:14.956739 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.952293 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" podStartSLOduration=2.283578556 podStartE2EDuration="6.952274476s" totalImagesPullingTime="4.66869592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:14.950092149 +0000 UTC m=+404.291675777" watchObservedRunningTime="2026-07-31 20:01:14.952274476 +0000 UTC m=+404.293858104" Jul 31 20:01:14.970481 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:14.970424 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" podStartSLOduration=1.232591988 podStartE2EDuration="3.970403908s" totalImagesPullingTime="2.73781192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:14.968194099 +0000 UTC m=+404.309777721" watchObservedRunningTime="2026-07-31 20:01:14.970403908 +0000 UTC m=+404.311987536" Jul 31 20:01:16.944286 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:16.944251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" event={"ID":"7eaae507-624a-42c3-9f2f-46f5a0a66c7a","Type":"ContainerStarted","Data":"3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867"} Jul 31 20:01:16.944662 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:16.944339 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:16.961504 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:16.961456 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" podStartSLOduration=1.910533612 podStartE2EDuration="3.961442147s" totalImagesPullingTime="2.050908535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:16.959412773 +0000 UTC m=+406.300996402" watchObservedRunningTime="2026-07-31 20:01:16.961442147 +0000 UTC m=+406.303025774" Jul 31 20:01:19.903889 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:19.903859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-d8xgh" Jul 31 20:01:25.940343 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:25.940286 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-qnwrs" Jul 31 20:01:25.940745 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:25.940492 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:27.901858 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.901823 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5"] Jul 31 20:01:27.902357 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.901998 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" containerName="manager" containerID="cri-o://f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896" gracePeriod=2 Jul 31 20:01:27.911664 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.911632 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5"] Jul 31 20:01:27.916894 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.916790 2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"31485a40-892c-4024-b9bf-08c5e4d7ec78\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-f5fk5\": pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:27.932992 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.932753 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8"] Jul 31 20:01:27.934144 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.934061 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" containerName="manager" containerID="cri-o://3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867" gracePeriod=2 Jul 31 20:01:27.937436 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.936516 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:27.938534 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.938329 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:27.938663 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.938538 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:27.939642 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.939563 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" containerName="manager" Jul 31 20:01:27.939755 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.939646 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" containerName="manager" Jul 31 20:01:27.939816 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.939779 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" containerName="manager" Jul 31 20:01:27.943380 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.943359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:27.943561 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.943548 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8"] Jul 31 20:01:27.943689 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.943667 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:27.946787 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.946752 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:27.949067 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.949041 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:27.951078 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.951052 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:27.953367 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.953268 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:27.957094 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.957068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:27.960034 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.960011 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw"] Jul 31 20:01:27.960324 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.960300 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" containerName="manager" Jul 31 20:01:27.960394 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.960328 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" containerName="manager" Jul 31 20:01:27.960394 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.960393 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" containerName="manager" Jul 31 20:01:27.964322 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.964287 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:27.977118 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:27.977083 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw"] Jul 31 20:01:27.980619 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:27.980588 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:28.001906 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.001872 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:28.035591 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.035556 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.035748 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.035613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrvk\" (UniqueName: \"kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.099684 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.099644 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:01:28.102911 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.102884 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.105116 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.105086 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:28.116263 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.116206 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:01:28.134186 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.134144 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:28.136518 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.136494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.136619 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.136535 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrvk\" (UniqueName: \"kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.136619 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.136567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz58\" (UniqueName: \"kubernetes.io/projected/f8a156ed-c678-4dde-86b6-7e81aa6fe149-kube-api-access-bsz58\") pod \"limitador-operator-controller-manager-6c549b6d85-gjzfw\" (UID: \"f8a156ed-c678-4dde-86b6-7e81aa6fe149\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:28.137029 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.136875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.148454 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.148420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrvk\" (UniqueName: \"kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk\") pod \"kuadrant-operator-controller-manager-58fc56757c-kvbms\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.184159 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.184133 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:28.186454 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.186436 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:28.186528 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.186451 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:28.188348 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.188327 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:28.190534 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.190505 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:28.192401 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:28.192375 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:28.237805 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.237772 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.237957 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.237828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz58\" (UniqueName: \"kubernetes.io/projected/f8a156ed-c678-4dde-86b6-7e81aa6fe149-kube-api-access-bsz58\") pod \"limitador-operator-controller-manager-6c549b6d85-gjzfw\" (UID: \"f8a156ed-c678-4dde-86b6-7e81aa6fe149\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:28.237957 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.237847 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blr4z\" (UniqueName: \"kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.248443 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.248415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz58\" (UniqueName: \"kubernetes.io/projected/f8a156ed-c678-4dde-86b6-7e81aa6fe149-kube-api-access-bsz58\") pod \"limitador-operator-controller-manager-6c549b6d85-gjzfw\" (UID: \"f8a156ed-c678-4dde-86b6-7e81aa6fe149\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:28.324396 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.324361 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:28.330297 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.330276 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:28.338417 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.338389 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdk5c\" (UniqueName: \"kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c\") pod \"31485a40-892c-4024-b9bf-08c5e4d7ec78\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " Jul 31 20:01:28.338498 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.338478 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf927\" (UniqueName: \"kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927\") pod \"7eaae507-624a-42c3-9f2f-46f5a0a66c7a\" (UID: \"7eaae507-624a-42c3-9f2f-46f5a0a66c7a\") " Jul 31 20:01:28.338544 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.338528 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume\") pod \"31485a40-892c-4024-b9bf-08c5e4d7ec78\" (UID: \"31485a40-892c-4024-b9bf-08c5e4d7ec78\") " Jul 31 20:01:28.338712 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.338692 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.338797 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.338779 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blr4z\" (UniqueName: \"kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.339067 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.339021 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "31485a40-892c-4024-b9bf-08c5e4d7ec78" (UID: "31485a40-892c-4024-b9bf-08c5e4d7ec78"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:01:28.339275 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.339252 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31485a40-892c-4024-b9bf-08c5e4d7ec78-extensions-socket-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:01:28.339730 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.339659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.340831 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.340808 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c" (OuterVolumeSpecName: "kube-api-access-jdk5c") pod "31485a40-892c-4024-b9bf-08c5e4d7ec78" (UID: "31485a40-892c-4024-b9bf-08c5e4d7ec78"). InnerVolumeSpecName "kube-api-access-jdk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:01:28.341152 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.341130 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927" (OuterVolumeSpecName: "kube-api-access-pf927") pod "7eaae507-624a-42c3-9f2f-46f5a0a66c7a" (UID: "7eaae507-624a-42c3-9f2f-46f5a0a66c7a"). InnerVolumeSpecName "kube-api-access-pf927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:01:28.353581 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.353535 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blr4z\" (UniqueName: \"kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z\") pod \"kuadrant-operator-controller-manager-68cf589589-hrmhb\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.417917 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.417607 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:28.440142 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.440088 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf927\" (UniqueName: \"kubernetes.io/projected/7eaae507-624a-42c3-9f2f-46f5a0a66c7a-kube-api-access-pf927\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:01:28.440142 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.440126 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdk5c\" (UniqueName: \"kubernetes.io/projected/31485a40-892c-4024-b9bf-08c5e4d7ec78-kube-api-access-jdk5c\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:01:28.481973 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.481792 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:28.484182 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:28.484139 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10efa1d3_e79f_4884_8d39_8bd4da33e38f.slice/crio-773c07053051f0d64bd2235e2c9bdc9faf4c8290585ffa9d5529398d541754f0 WatchSource:0}: Error finding container 773c07053051f0d64bd2235e2c9bdc9faf4c8290585ffa9d5529398d541754f0: Status 404 returned error can't find the container with id 773c07053051f0d64bd2235e2c9bdc9faf4c8290585ffa9d5529398d541754f0 Jul 31 20:01:28.509704 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.509676 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw"] Jul 31 20:01:28.520100 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:28.520065 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a156ed_c678_4dde_86b6_7e81aa6fe149.slice/crio-2dfe3970895139aa2db0e8514a8c2fbb203020ed8c034cb888b77da1de6f04eb WatchSource:0}: Error finding container 2dfe3970895139aa2db0e8514a8c2fbb203020ed8c034cb888b77da1de6f04eb: Status 404 returned error can't find the container with id 2dfe3970895139aa2db0e8514a8c2fbb203020ed8c034cb888b77da1de6f04eb Jul 31 20:01:28.584072 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:28.584038 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:01:28.586188 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:28.586160 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86073eaf_f62a_4376_92c2_e01644d1645e.slice/crio-2a6840aa0d96911c22b3ed658346096a0a73c13b1f08bff2fd4c71f8894548c7 WatchSource:0}: Error finding container 2a6840aa0d96911c22b3ed658346096a0a73c13b1f08bff2fd4c71f8894548c7: Status 404 returned error can't find the container with id 2a6840aa0d96911c22b3ed658346096a0a73c13b1f08bff2fd4c71f8894548c7 Jul 31 20:01:29.005671 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.005636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" event={"ID":"10efa1d3-e79f-4884-8d39-8bd4da33e38f","Type":"ContainerStarted","Data":"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a"} Jul 31 20:01:29.006134 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.005684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" event={"ID":"10efa1d3-e79f-4884-8d39-8bd4da33e38f","Type":"ContainerStarted","Data":"773c07053051f0d64bd2235e2c9bdc9faf4c8290585ffa9d5529398d541754f0"} Jul 31 20:01:29.006134 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.005748 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:29.007174 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.007152 2556 generic.go:356] "Generic (PLEG): container finished" podID="31485a40-892c-4024-b9bf-08c5e4d7ec78" containerID="f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896" exitCode=0 Jul 31 20:01:29.007249 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.007191 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:29.007299 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.007251 2556 scope.go:117] "RemoveContainer" containerID="f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896" Jul 31 20:01:29.007670 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.007644 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:29.009105 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.009077 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" event={"ID":"86073eaf-f62a-4376-92c2-e01644d1645e","Type":"ContainerStarted","Data":"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd"} Jul 31 20:01:29.009214 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.009131 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" event={"ID":"86073eaf-f62a-4376-92c2-e01644d1645e","Type":"ContainerStarted","Data":"2a6840aa0d96911c22b3ed658346096a0a73c13b1f08bff2fd4c71f8894548c7"} Jul 31 20:01:29.009214 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.009173 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:29.010990 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.010964 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" event={"ID":"f8a156ed-c678-4dde-86b6-7e81aa6fe149","Type":"ContainerStarted","Data":"d2f482ce4715c69a8fa096fbe56525eacda4b4e6a63ac6a168eae16b66669725"} Jul 31 20:01:29.011103 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.011015 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" event={"ID":"f8a156ed-c678-4dde-86b6-7e81aa6fe149","Type":"ContainerStarted","Data":"2dfe3970895139aa2db0e8514a8c2fbb203020ed8c034cb888b77da1de6f04eb"} Jul 31 20:01:29.011103 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.011045 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:29.012333 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.012297 2556 generic.go:356] "Generic (PLEG): container finished" podID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" containerID="3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867" exitCode=0 Jul 31 20:01:29.012407 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.012349 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:29.017570 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.017550 2556 scope.go:117] "RemoveContainer" containerID="f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896" Jul 31 20:01:29.017826 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.017811 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896\": container with ID starting with f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896 not found: ID does not exist" containerID="f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896" Jul 31 20:01:29.017870 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.017833 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896"} err="failed to get container status \"f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896\": rpc error: code = NotFound desc = could not find container \"f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896\": container with ID starting with f8b27110a89e217781729e27b5dd4e5914c1ff752de93f98e778b894599e9896 not found: ID does not exist" Jul 31 20:01:29.017870 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.017849 2556 scope.go:117] "RemoveContainer" containerID="3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867" Jul 31 20:01:29.025673 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.025640 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:29.025875 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.025841 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" podStartSLOduration=2.025828586 podStartE2EDuration="2.025828586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:29.023545197 +0000 UTC m=+418.365128824" watchObservedRunningTime="2026-07-31 20:01:29.025828586 +0000 UTC m=+418.367412251" Jul 31 20:01:29.026274 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.026256 2556 scope.go:117] "RemoveContainer" containerID="3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867" Jul 31 20:01:29.026607 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.026587 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867\": container with ID starting with 3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867 not found: ID does not exist" containerID="3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867" Jul 31 20:01:29.026670 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.026617 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867"} err="failed to get container status \"3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867\": rpc error: code = NotFound desc = could not find container \"3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867\": container with ID starting with 3f8fa737d309bd3af408ed10969aaf32be6da5b5135a66be7d015bb04d8a7867 not found: ID does not exist" Jul 31 20:01:29.027433 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.027405 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-bb8b8\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bb8b8" Jul 31 20:01:29.029182 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:29.029156 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-f5fk5\" is forbidden: User \"system:node:ip-10-0-143-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-178.ec2.internal' and this object" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f5fk5" Jul 31 20:01:29.048663 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.048605 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" podStartSLOduration=2.048589398 podStartE2EDuration="2.048589398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:29.047659941 +0000 UTC m=+418.389243569" watchObservedRunningTime="2026-07-31 20:01:29.048589398 +0000 UTC m=+418.390173024" Jul 31 20:01:29.068585 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.068528 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" podStartSLOduration=1.068512826 podStartE2EDuration="1.068512826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:01:29.066677762 +0000 UTC m=+418.408261380" watchObservedRunningTime="2026-07-31 20:01:29.068512826 +0000 UTC m=+418.410096453" Jul 31 20:01:29.167328 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.167277 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31485a40-892c-4024-b9bf-08c5e4d7ec78" path="/var/lib/kubelet/pods/31485a40-892c-4024-b9bf-08c5e4d7ec78/volumes" Jul 31 20:01:29.167624 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:29.167610 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eaae507-624a-42c3-9f2f-46f5a0a66c7a" path="/var/lib/kubelet/pods/7eaae507-624a-42c3-9f2f-46f5a0a66c7a/volumes" Jul 31 20:01:40.021298 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.021265 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:01:40.021780 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.021361 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:40.021780 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.021453 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gjzfw" Jul 31 20:01:40.099098 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.099070 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:40.099358 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.099328 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" podUID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" containerName="manager" containerID="cri-o://212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a" gracePeriod=10 Jul 31 20:01:40.337690 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.337668 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:40.441133 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.441106 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqrvk\" (UniqueName: \"kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk\") pod \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " Jul 31 20:01:40.441287 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.441163 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume\") pod \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\" (UID: \"10efa1d3-e79f-4884-8d39-8bd4da33e38f\") " Jul 31 20:01:40.441584 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.441560 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10efa1d3-e79f-4884-8d39-8bd4da33e38f" (UID: "10efa1d3-e79f-4884-8d39-8bd4da33e38f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:01:40.443242 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.443215 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk" (OuterVolumeSpecName: "kube-api-access-dqrvk") pod "10efa1d3-e79f-4884-8d39-8bd4da33e38f" (UID: "10efa1d3-e79f-4884-8d39-8bd4da33e38f"). InnerVolumeSpecName "kube-api-access-dqrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:01:40.542274 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.542245 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrvk\" (UniqueName: \"kubernetes.io/projected/10efa1d3-e79f-4884-8d39-8bd4da33e38f-kube-api-access-dqrvk\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:01:40.542274 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:40.542270 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10efa1d3-e79f-4884-8d39-8bd4da33e38f-extensions-socket-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:01:41.073297 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.073261 2556 generic.go:356] "Generic (PLEG): container finished" podID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" containerID="212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a" exitCode=0 Jul 31 20:01:41.073297 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.073298 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" Jul 31 20:01:41.073789 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.073349 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" event={"ID":"10efa1d3-e79f-4884-8d39-8bd4da33e38f","Type":"ContainerDied","Data":"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a"} Jul 31 20:01:41.073789 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.073420 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms" event={"ID":"10efa1d3-e79f-4884-8d39-8bd4da33e38f","Type":"ContainerDied","Data":"773c07053051f0d64bd2235e2c9bdc9faf4c8290585ffa9d5529398d541754f0"} Jul 31 20:01:41.073789 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.073453 2556 scope.go:117] "RemoveContainer" containerID="212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a" Jul 31 20:01:41.082411 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.082388 2556 scope.go:117] "RemoveContainer" containerID="212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a" Jul 31 20:01:41.082661 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:01:41.082644 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a\": container with ID starting with 212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a not found: ID does not exist" containerID="212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a" Jul 31 20:01:41.082710 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.082670 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a"} err="failed to get container status \"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a\": rpc error: code = NotFound desc = could not find container \"212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a\": container with ID starting with 212e6bba39d452feb05888f9609fe9f932e5fea7ee25f7fcd05e554780b1294a not found: ID does not exist" Jul 31 20:01:41.098787 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.098765 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:41.105322 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.105288 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-kvbms"] Jul 31 20:01:41.165199 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:41.165177 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" path="/var/lib/kubelet/pods/10efa1d3-e79f-4884-8d39-8bd4da33e38f/volumes" Jul 31 20:01:59.070924 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.070840 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:01:59.071428 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.071165 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" containerName="manager" Jul 31 20:01:59.071428 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.071178 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" containerName="manager" Jul 31 20:01:59.071428 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.071232 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="10efa1d3-e79f-4884-8d39-8bd4da33e38f" containerName="manager" Jul 31 20:01:59.074297 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.074274 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.076648 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.076612 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 20:01:59.076770 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.076649 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-br4jn\"" Jul 31 20:01:59.083075 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.083023 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:01:59.159531 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.159500 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.159531 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.159533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9n6m\" (UniqueName: \"kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.165651 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.165625 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:01:59.260039 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.259995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.260226 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.260065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9n6m\" (UniqueName: \"kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.260842 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.260808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.268165 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.268139 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9n6m\" (UniqueName: \"kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m\") pod \"limitador-limitador-859c4d96d6-4rqvv\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.387931 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.387859 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:01:59.542986 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:01:59.542958 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:01:59.545055 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:01:59.545023 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925d4033_1426_4193_97b4_5348aa5d3f15.slice/crio-a18c5b89b96fe3fef65a444f3e5934254212a24335a47df939f7ba7758a8fc65 WatchSource:0}: Error finding container a18c5b89b96fe3fef65a444f3e5934254212a24335a47df939f7ba7758a8fc65: Status 404 returned error can't find the container with id a18c5b89b96fe3fef65a444f3e5934254212a24335a47df939f7ba7758a8fc65 Jul 31 20:02:00.046227 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.046192 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:00.050578 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.050560 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:00.052951 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.052863 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mt98w\"" Jul 31 20:02:00.055542 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.055517 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:00.066145 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.066118 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78s9\" (UniqueName: \"kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9\") pod \"authorino-c548b5d7d-r84n4\" (UID: \"83319543-badd-48b9-9487-b0f846cede5e\") " pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:00.166605 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.166558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78s9\" (UniqueName: \"kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9\") pod \"authorino-c548b5d7d-r84n4\" (UID: \"83319543-badd-48b9-9487-b0f846cede5e\") " pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:00.167031 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.166954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" event={"ID":"925d4033-1426-4193-97b4-5348aa5d3f15","Type":"ContainerStarted","Data":"a18c5b89b96fe3fef65a444f3e5934254212a24335a47df939f7ba7758a8fc65"} Jul 31 20:02:00.174696 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.174667 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78s9\" (UniqueName: \"kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9\") pod \"authorino-c548b5d7d-r84n4\" (UID: \"83319543-badd-48b9-9487-b0f846cede5e\") " pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:00.361390 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.361359 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:00.484798 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:00.484765 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:00.487486 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:02:00.487444 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83319543_badd_48b9_9487_b0f846cede5e.slice/crio-ecacf49be49320c863f8c1746d9f9afce19573a7368f8f93fe271b08d1f126b0 WatchSource:0}: Error finding container ecacf49be49320c863f8c1746d9f9afce19573a7368f8f93fe271b08d1f126b0: Status 404 returned error can't find the container with id ecacf49be49320c863f8c1746d9f9afce19573a7368f8f93fe271b08d1f126b0 Jul 31 20:02:01.172978 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:01.172943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r84n4" event={"ID":"83319543-badd-48b9-9487-b0f846cede5e","Type":"ContainerStarted","Data":"ecacf49be49320c863f8c1746d9f9afce19573a7368f8f93fe271b08d1f126b0"} Jul 31 20:02:04.190922 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:04.190873 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" event={"ID":"925d4033-1426-4193-97b4-5348aa5d3f15","Type":"ContainerStarted","Data":"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a"} Jul 31 20:02:04.191406 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:04.190962 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:02:04.207498 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:04.207433 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" podStartSLOduration=1.492398149 podStartE2EDuration="5.207416077s" totalImagesPullingTime="3.715017928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:02:04.205137566 +0000 UTC m=+453.546721194" watchObservedRunningTime="2026-07-31 20:02:04.207416077 +0000 UTC m=+453.548999705" Jul 31 20:02:05.196965 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:05.196876 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r84n4" event={"ID":"83319543-badd-48b9-9487-b0f846cede5e","Type":"ContainerStarted","Data":"689b206c2bc7358d48946af8ad5ecb38f0217a330130ce90b8e0776ea5f6eff6"} Jul 31 20:02:05.213457 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:05.213408 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-r84n4" podStartSLOduration=0.890585529 podStartE2EDuration="5.213393021s" totalImagesPullingTime="4.322807492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:02:05.211066842 +0000 UTC m=+454.552650469" watchObservedRunningTime="2026-07-31 20:02:05.213393021 +0000 UTC m=+454.554976648" Jul 31 20:02:15.198418 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:15.198385 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:02:16.125868 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.125831 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:02:16.126063 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.126026 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" podUID="925d4033-1426-4193-97b4-5348aa5d3f15" containerName="limitador" containerID="cri-o://59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a" gracePeriod=30 Jul 31 20:02:16.659895 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.659874 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:02:16.692865 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.692808 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file\") pod \"925d4033-1426-4193-97b4-5348aa5d3f15\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " Jul 31 20:02:16.692865 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.692841 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9n6m\" (UniqueName: \"kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m\") pod \"925d4033-1426-4193-97b4-5348aa5d3f15\" (UID: \"925d4033-1426-4193-97b4-5348aa5d3f15\") " Jul 31 20:02:16.693163 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.693142 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file" (OuterVolumeSpecName: "config-file") pod "925d4033-1426-4193-97b4-5348aa5d3f15" (UID: "925d4033-1426-4193-97b4-5348aa5d3f15"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:02:16.694888 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.694861 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m" (OuterVolumeSpecName: "kube-api-access-l9n6m") pod "925d4033-1426-4193-97b4-5348aa5d3f15" (UID: "925d4033-1426-4193-97b4-5348aa5d3f15"). InnerVolumeSpecName "kube-api-access-l9n6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:02:16.794275 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.794251 2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/925d4033-1426-4193-97b4-5348aa5d3f15-config-file\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:16.794275 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:16.794272 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9n6m\" (UniqueName: \"kubernetes.io/projected/925d4033-1426-4193-97b4-5348aa5d3f15-kube-api-access-l9n6m\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:17.257754 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.257721 2556 generic.go:356] "Generic (PLEG): container finished" podID="925d4033-1426-4193-97b4-5348aa5d3f15" containerID="59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a" exitCode=0 Jul 31 20:02:17.257918 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.257758 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" Jul 31 20:02:17.257918 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.257814 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" event={"ID":"925d4033-1426-4193-97b4-5348aa5d3f15","Type":"ContainerDied","Data":"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a"} Jul 31 20:02:17.257918 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.257859 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-4rqvv" event={"ID":"925d4033-1426-4193-97b4-5348aa5d3f15","Type":"ContainerDied","Data":"a18c5b89b96fe3fef65a444f3e5934254212a24335a47df939f7ba7758a8fc65"} Jul 31 20:02:17.257918 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.257880 2556 scope.go:117] "RemoveContainer" containerID="59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a" Jul 31 20:02:17.266239 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.266222 2556 scope.go:117] "RemoveContainer" containerID="59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a" Jul 31 20:02:17.266540 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:17.266515 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a\": container with ID starting with 59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a not found: ID does not exist" containerID="59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a" Jul 31 20:02:17.266633 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.266546 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a"} err="failed to get container status \"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a\": rpc error: code = NotFound desc = could not find container \"59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a\": container with ID starting with 59f2dbcbdb73a2e821a289dde2c51446144df211191cd58b84e8412ccb3de39a not found: ID does not exist" Jul 31 20:02:17.273622 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.273599 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:02:17.277198 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:17.277178 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-4rqvv"] Jul 31 20:02:19.165597 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:19.165564 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925d4033-1426-4193-97b4-5348aa5d3f15" path="/var/lib/kubelet/pods/925d4033-1426-4193-97b4-5348aa5d3f15/volumes" Jul 31 20:02:20.869388 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.869244 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg"] Jul 31 20:02:20.869956 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.869932 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925d4033-1426-4193-97b4-5348aa5d3f15" containerName="limitador" Jul 31 20:02:20.870055 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.869959 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="925d4033-1426-4193-97b4-5348aa5d3f15" containerName="limitador" Jul 31 20:02:20.870157 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.870144 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="925d4033-1426-4193-97b4-5348aa5d3f15" containerName="limitador" Jul 31 20:02:20.875359 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.875330 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:20.877957 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.877933 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 20:02:20.878566 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.878513 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kkmcr\"" Jul 31 20:02:20.878566 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.878519 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 31 20:02:20.878703 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.878522 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 31 20:02:20.879907 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.879882 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg"] Jul 31 20:02:20.927389 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.927353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrcc6\" (UniqueName: \"kubernetes.io/projected/59017ab3-9972-49fd-9378-dcdb355784e9-kube-api-access-vrcc6\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:20.927565 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:20.927409 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59017ab3-9972-49fd-9378-dcdb355784e9-data\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.028800 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.028762 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrcc6\" (UniqueName: \"kubernetes.io/projected/59017ab3-9972-49fd-9378-dcdb355784e9-kube-api-access-vrcc6\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.028936 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.028817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59017ab3-9972-49fd-9378-dcdb355784e9-data\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.029206 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.029186 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59017ab3-9972-49fd-9378-dcdb355784e9-data\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.036571 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.036543 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrcc6\" (UniqueName: \"kubernetes.io/projected/59017ab3-9972-49fd-9378-dcdb355784e9-kube-api-access-vrcc6\") pod \"postgres-69fb4594cf-tzbhg\" (UID: \"59017ab3-9972-49fd-9378-dcdb355784e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.187240 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.187166 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:21.315935 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:21.315879 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg"] Jul 31 20:02:21.317996 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:02:21.317962 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59017ab3_9972_49fd_9378_dcdb355784e9.slice/crio-697b00748596e4a723f3d29f6e409626d257f3a3de0e600600acd1621fe61a2c WatchSource:0}: Error finding container 697b00748596e4a723f3d29f6e409626d257f3a3de0e600600acd1621fe61a2c: Status 404 returned error can't find the container with id 697b00748596e4a723f3d29f6e409626d257f3a3de0e600600acd1621fe61a2c Jul 31 20:02:22.288395 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:22.288358 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" event={"ID":"59017ab3-9972-49fd-9378-dcdb355784e9","Type":"ContainerStarted","Data":"697b00748596e4a723f3d29f6e409626d257f3a3de0e600600acd1621fe61a2c"} Jul 31 20:02:27.318717 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:27.318676 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" event={"ID":"59017ab3-9972-49fd-9378-dcdb355784e9","Type":"ContainerStarted","Data":"b1e7b89afc49c2c09765bd2596eba5af982ddb46320005cc6ee3855001607e91"} Jul 31 20:02:27.319054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:27.318739 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:27.336062 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:27.336013 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" podStartSLOduration=2.243828274 podStartE2EDuration="7.335997837s" totalImagesPullingTime="5.092169563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:02:27.334111126 +0000 UTC m=+476.675694753" watchObservedRunningTime="2026-07-31 20:02:27.335997837 +0000 UTC m=+476.677581464" Jul 31 20:02:33.353634 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:33.353605 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tzbhg" Jul 31 20:02:33.909954 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:33.909914 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-fm4r6"] Jul 31 20:02:33.916172 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:33.916148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:33.916299 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:33.916234 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-fm4r6"] Jul 31 20:02:34.036988 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.036956 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5vc\" (UniqueName: \"kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc\") pod \"authorino-5dc7ccb5f-fm4r6\" (UID: \"7ee185a5-2eac-4028-8cfc-6d4676ae5db4\") " pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:34.138308 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.138270 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5vc\" (UniqueName: \"kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc\") pod \"authorino-5dc7ccb5f-fm4r6\" (UID: \"7ee185a5-2eac-4028-8cfc-6d4676ae5db4\") " pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:34.141614 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.141585 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-fm4r6"] Jul 31 20:02:34.141824 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:34.141806 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xd5vc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" podUID="7ee185a5-2eac-4028-8cfc-6d4676ae5db4" Jul 31 20:02:34.148759 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.148731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5vc\" (UniqueName: \"kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc\") pod \"authorino-5dc7ccb5f-fm4r6\" (UID: \"7ee185a5-2eac-4028-8cfc-6d4676ae5db4\") " pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:34.353782 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.353748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:34.361951 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.361927 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:34.441163 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.441134 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd5vc\" (UniqueName: \"kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc\") pod \"7ee185a5-2eac-4028-8cfc-6d4676ae5db4\" (UID: \"7ee185a5-2eac-4028-8cfc-6d4676ae5db4\") " Jul 31 20:02:34.443166 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.443141 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc" (OuterVolumeSpecName: "kube-api-access-xd5vc") pod "7ee185a5-2eac-4028-8cfc-6d4676ae5db4" (UID: "7ee185a5-2eac-4028-8cfc-6d4676ae5db4"). InnerVolumeSpecName "kube-api-access-xd5vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:02:34.542041 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:34.542005 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5vc\" (UniqueName: \"kubernetes.io/projected/7ee185a5-2eac-4028-8cfc-6d4676ae5db4-kube-api-access-xd5vc\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:35.358327 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:35.358283 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-fm4r6" Jul 31 20:02:35.382412 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:35.382381 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-fm4r6"] Jul 31 20:02:35.387405 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:35.387374 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-fm4r6"] Jul 31 20:02:37.165600 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:37.165567 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee185a5-2eac-4028-8cfc-6d4676ae5db4" path="/var/lib/kubelet/pods/7ee185a5-2eac-4028-8cfc-6d4676ae5db4/volumes" Jul 31 20:02:38.716321 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:38.716282 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:38.716675 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:38.716451 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-r84n4" podUID="83319543-badd-48b9-9487-b0f846cede5e" containerName="authorino" containerID="cri-o://689b206c2bc7358d48946af8ad5ecb38f0217a330130ce90b8e0776ea5f6eff6" gracePeriod=30 Jul 31 20:02:39.380344 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.380289 2556 generic.go:356] "Generic (PLEG): container finished" podID="83319543-badd-48b9-9487-b0f846cede5e" containerID="689b206c2bc7358d48946af8ad5ecb38f0217a330130ce90b8e0776ea5f6eff6" exitCode=0 Jul 31 20:02:39.380475 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.380350 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r84n4" event={"ID":"83319543-badd-48b9-9487-b0f846cede5e","Type":"ContainerDied","Data":"689b206c2bc7358d48946af8ad5ecb38f0217a330130ce90b8e0776ea5f6eff6"} Jul 31 20:02:39.522628 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.522600 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:39.684745 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.684663 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t78s9\" (UniqueName: \"kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9\") pod \"83319543-badd-48b9-9487-b0f846cede5e\" (UID: \"83319543-badd-48b9-9487-b0f846cede5e\") " Jul 31 20:02:39.686750 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.686719 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9" (OuterVolumeSpecName: "kube-api-access-t78s9") pod "83319543-badd-48b9-9487-b0f846cede5e" (UID: "83319543-badd-48b9-9487-b0f846cede5e"). InnerVolumeSpecName "kube-api-access-t78s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:02:39.785629 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:39.785591 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78s9\" (UniqueName: \"kubernetes.io/projected/83319543-badd-48b9-9487-b0f846cede5e-kube-api-access-t78s9\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:40.387077 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:40.387041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-r84n4" event={"ID":"83319543-badd-48b9-9487-b0f846cede5e","Type":"ContainerDied","Data":"ecacf49be49320c863f8c1746d9f9afce19573a7368f8f93fe271b08d1f126b0"} Jul 31 20:02:40.387077 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:40.387074 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-r84n4" Jul 31 20:02:40.387289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:40.387092 2556 scope.go:117] "RemoveContainer" containerID="689b206c2bc7358d48946af8ad5ecb38f0217a330130ce90b8e0776ea5f6eff6" Jul 31 20:02:40.408298 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:40.408271 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:40.413289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:40.413263 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-r84n4"] Jul 31 20:02:41.166323 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:41.166288 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83319543-badd-48b9-9487-b0f846cede5e" path="/var/lib/kubelet/pods/83319543-badd-48b9-9487-b0f846cede5e/volumes" Jul 31 20:02:43.397924 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.397887 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:02:43.398306 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.398219 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83319543-badd-48b9-9487-b0f846cede5e" containerName="authorino" Jul 31 20:02:43.398306 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.398235 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="83319543-badd-48b9-9487-b0f846cede5e" containerName="authorino" Jul 31 20:02:43.398418 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.398329 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="83319543-badd-48b9-9487-b0f846cede5e" containerName="authorino" Jul 31 20:02:43.401479 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.401444 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.404026 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.404001 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gg4jb\"" Jul 31 20:02:43.404173 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.404155 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 20:02:43.404710 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.404695 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 20:02:43.413806 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.413782 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:02:43.513225 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.513199 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkzc\" (UniqueName: \"kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.513404 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.513241 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.559114 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.559084 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:02:43.561598 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.561578 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:43.571363 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.571338 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:02:43.613586 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.613563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.613704 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.613624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkzc\" (UniqueName: \"kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.613750 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:43.613720 2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 31 20:02:43.613785 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:43.613780 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert podName:d8813ddc-91ec-41bd-9f72-f272cf1730c5 nodeName:}" failed. No retries permitted until 2026-07-31 20:02:44.113761113 +0000 UTC m=+493.455344735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert") pod "maas-controller-85dff98fcb-cbxcl" (UID: "d8813ddc-91ec-41bd-9f72-f272cf1730c5") : secret "maas-controller-webhook-cert" not found Jul 31 20:02:43.621866 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.621846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkzc\" (UniqueName: \"kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:43.714551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.714459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:43.714551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.714505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dnj\" (UniqueName: \"kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:43.815464 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.815429 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:43.815603 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.815471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dnj\" (UniqueName: \"kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:43.815603 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:43.815589 2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 31 20:02:43.815699 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:43.815687 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert podName:c7744847-4a64-41b7-837b-33d8377fb500 nodeName:}" failed. No retries permitted until 2026-07-31 20:02:44.31566783 +0000 UTC m=+493.657251446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert") pod "maas-controller-777c587c75-wzp6s" (UID: "c7744847-4a64-41b7-837b-33d8377fb500") : secret "maas-controller-webhook-cert" not found Jul 31 20:02:43.825668 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:43.825638 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dnj\" (UniqueName: \"kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:44.117823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.117785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:44.120166 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.120134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") pod \"maas-controller-85dff98fcb-cbxcl\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:44.312948 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.312909 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:44.319558 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.319539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:44.321718 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.321699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") pod \"maas-controller-777c587c75-wzp6s\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:44.442351 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.442326 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:02:44.443892 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:02:44.443865 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8813ddc_91ec_41bd_9f72_f272cf1730c5.slice/crio-d5461d634ed5866e6575aea0c93a2cc2fc41489b2863bfce8fca6c0d4d5bdd08 WatchSource:0}: Error finding container d5461d634ed5866e6575aea0c93a2cc2fc41489b2863bfce8fca6c0d4d5bdd08: Status 404 returned error can't find the container with id d5461d634ed5866e6575aea0c93a2cc2fc41489b2863bfce8fca6c0d4d5bdd08 Jul 31 20:02:44.471377 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.471352 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:44.596994 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:44.596970 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:02:44.598825 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:02:44.598798 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7744847_4a64_41b7_837b_33d8377fb500.slice/crio-c1baecc17021d7756a74e022d05b6ee5df984754335fea0c1aef9132b9113666 WatchSource:0}: Error finding container c1baecc17021d7756a74e022d05b6ee5df984754335fea0c1aef9132b9113666: Status 404 returned error can't find the container with id c1baecc17021d7756a74e022d05b6ee5df984754335fea0c1aef9132b9113666 Jul 31 20:02:45.417669 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:45.417628 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-wzp6s" event={"ID":"c7744847-4a64-41b7-837b-33d8377fb500","Type":"ContainerStarted","Data":"c1baecc17021d7756a74e022d05b6ee5df984754335fea0c1aef9132b9113666"} Jul 31 20:02:45.420923 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:45.420846 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" event={"ID":"d8813ddc-91ec-41bd-9f72-f272cf1730c5","Type":"ContainerStarted","Data":"d5461d634ed5866e6575aea0c93a2cc2fc41489b2863bfce8fca6c0d4d5bdd08"} Jul 31 20:02:47.433449 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.433417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-wzp6s" event={"ID":"c7744847-4a64-41b7-837b-33d8377fb500","Type":"ContainerStarted","Data":"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9"} Jul 31 20:02:47.433925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.433476 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:47.434891 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.434865 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" event={"ID":"d8813ddc-91ec-41bd-9f72-f272cf1730c5","Type":"ContainerStarted","Data":"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733"} Jul 31 20:02:47.434988 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.434980 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:47.455541 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.455497 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777c587c75-wzp6s" podStartSLOduration=2.0663534 podStartE2EDuration="4.455482388s" totalImagesPullingTime="2.389128988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:02:47.454438342 +0000 UTC m=+496.796021969" watchObservedRunningTime="2026-07-31 20:02:47.455482388 +0000 UTC m=+496.797066014" Jul 31 20:02:47.476329 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:47.476273 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" podStartSLOduration=1.9384120870000001 podStartE2EDuration="4.476260579s" totalImagesPullingTime="2.537848492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:02:47.47577863 +0000 UTC m=+496.817362289" watchObservedRunningTime="2026-07-31 20:02:47.476260579 +0000 UTC m=+496.817844205" Jul 31 20:02:58.442067 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.442035 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:58.442450 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.442124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:02:58.501307 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.501269 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:02:58.501565 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.501531 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" podUID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" containerName="manager" containerID="cri-o://5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733" gracePeriod=30 Jul 31 20:02:58.743594 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.743570 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:58.941993 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.941956 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") pod \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " Jul 31 20:02:58.941993 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.942002 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvkzc\" (UniqueName: \"kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc\") pod \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\" (UID: \"d8813ddc-91ec-41bd-9f72-f272cf1730c5\") " Jul 31 20:02:58.944203 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.944175 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d8813ddc-91ec-41bd-9f72-f272cf1730c5" (UID: "d8813ddc-91ec-41bd-9f72-f272cf1730c5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:02:58.944343 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:58.944265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc" (OuterVolumeSpecName: "kube-api-access-gvkzc") pod "d8813ddc-91ec-41bd-9f72-f272cf1730c5" (UID: "d8813ddc-91ec-41bd-9f72-f272cf1730c5"). InnerVolumeSpecName "kube-api-access-gvkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:02:59.042974 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.042901 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8813ddc-91ec-41bd-9f72-f272cf1730c5-webhook-cert\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:59.042974 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.042928 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkzc\" (UniqueName: \"kubernetes.io/projected/d8813ddc-91ec-41bd-9f72-f272cf1730c5-kube-api-access-gvkzc\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:02:59.494754 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.494720 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" containerID="5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733" exitCode=0 Jul 31 20:02:59.495205 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.494773 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" Jul 31 20:02:59.495205 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.494793 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" event={"ID":"d8813ddc-91ec-41bd-9f72-f272cf1730c5","Type":"ContainerDied","Data":"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733"} Jul 31 20:02:59.495205 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.494840 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85dff98fcb-cbxcl" event={"ID":"d8813ddc-91ec-41bd-9f72-f272cf1730c5","Type":"ContainerDied","Data":"d5461d634ed5866e6575aea0c93a2cc2fc41489b2863bfce8fca6c0d4d5bdd08"} Jul 31 20:02:59.495205 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.494864 2556 scope.go:117] "RemoveContainer" containerID="5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733" Jul 31 20:02:59.503111 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.502934 2556 scope.go:117] "RemoveContainer" containerID="5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733" Jul 31 20:02:59.503200 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:02:59.503178 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733\": container with ID starting with 5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733 not found: ID does not exist" containerID="5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733" Jul 31 20:02:59.503240 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.503210 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733"} err="failed to get container status \"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733\": rpc error: code = NotFound desc = could not find container \"5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733\": container with ID starting with 5da5a72ec1a5bdb0110d1f3f717405d8d10d5fd2e21080a92c34e64d824ef733 not found: ID does not exist" Jul 31 20:02:59.510276 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.510252 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:02:59.513912 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:02:59.513892 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85dff98fcb-cbxcl"] Jul 31 20:03:01.165935 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:01.165906 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" path="/var/lib/kubelet/pods/d8813ddc-91ec-41bd-9f72-f272cf1730c5/volumes" Jul 31 20:03:21.407131 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.407094 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c4944758f-r5l5m"] Jul 31 20:03:21.407612 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.407416 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" containerName="manager" Jul 31 20:03:21.407612 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.407427 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" containerName="manager" Jul 31 20:03:21.407612 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.407482 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8813ddc-91ec-41bd-9f72-f272cf1730c5" containerName="manager" Jul 31 20:03:21.414056 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.414037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.416610 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.416589 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 31 20:03:21.416745 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.416685 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q2zm9\"" Jul 31 20:03:21.418433 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.418406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c4944758f-r5l5m"] Jul 31 20:03:21.512076 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.512042 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.512238 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.512089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbt5\" (UniqueName: \"kubernetes.io/projected/a9883ae5-2328-48fe-aa30-c3550ea96097-kube-api-access-wtbt5\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.512238 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.512128 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9883ae5-2328-48fe-aa30-c3550ea96097-openshift-service-ca\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.613520 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.613488 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.613682 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.613525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbt5\" (UniqueName: \"kubernetes.io/projected/a9883ae5-2328-48fe-aa30-c3550ea96097-kube-api-access-wtbt5\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.613682 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.613556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9883ae5-2328-48fe-aa30-c3550ea96097-openshift-service-ca\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.613682 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:03:21.613635 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 31 20:03:21.613786 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:03:21.613721 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls podName:a9883ae5-2328-48fe-aa30-c3550ea96097 nodeName:}" failed. No retries permitted until 2026-07-31 20:03:22.113703948 +0000 UTC m=+531.455287556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls") pod "maas-api-c4944758f-r5l5m" (UID: "a9883ae5-2328-48fe-aa30-c3550ea96097") : secret "maas-api-serving-cert" not found Jul 31 20:03:21.614252 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.614233 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9883ae5-2328-48fe-aa30-c3550ea96097-openshift-service-ca\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.621981 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.621956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbt5\" (UniqueName: \"kubernetes.io/projected/a9883ae5-2328-48fe-aa30-c3550ea96097-kube-api-access-wtbt5\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:21.898179 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.898149 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv"] Jul 31 20:03:21.901632 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.901618 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:21.904057 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.904037 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qn2b2\"" Jul 31 20:03:21.904162 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.904040 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 31 20:03:21.913775 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.913752 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv"] Jul 31 20:03:21.936877 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.936849 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2p7zz"] Jul 31 20:03:21.940071 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.940053 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:21.952506 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:21.952482 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2p7zz"] Jul 31 20:03:22.017541 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.017507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7n2\" (UniqueName: \"kubernetes.io/projected/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-kube-api-access-zr7n2\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.017541 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.017547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedaa52f-8b8e-462f-83e7-fcf362f38826-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.017848 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.017568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.017848 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.017647 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkds\" (UniqueName: \"kubernetes.io/projected/eedaa52f-8b8e-462f-83e7-fcf362f38826-kube-api-access-phkds\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.118083 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7n2\" (UniqueName: \"kubernetes.io/projected/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-kube-api-access-zr7n2\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.118083 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118088 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedaa52f-8b8e-462f-83e7-fcf362f38826-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.118396 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118110 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.118396 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkds\" (UniqueName: \"kubernetes.io/projected/eedaa52f-8b8e-462f-83e7-fcf362f38826-kube-api-access-phkds\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.118396 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118206 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:22.118901 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118882 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedaa52f-8b8e-462f-83e7-fcf362f38826-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.118945 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.118922 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.120563 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.120547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9883ae5-2328-48fe-aa30-c3550ea96097-maas-api-tls\") pod \"maas-api-c4944758f-r5l5m\" (UID: \"a9883ae5-2328-48fe-aa30-c3550ea96097\") " pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:22.125579 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.125557 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkds\" (UniqueName: \"kubernetes.io/projected/eedaa52f-8b8e-462f-83e7-fcf362f38826-kube-api-access-phkds\") pod \"payload-pre-processing-687dbb6cf6-b47wv\" (UID: \"eedaa52f-8b8e-462f-83e7-fcf362f38826\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.125765 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.125744 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7n2\" (UniqueName: \"kubernetes.io/projected/a2e3916c-31ab-4d2e-9bd7-19ec0eab3033-kube-api-access-zr7n2\") pod \"payload-processing-5cc6cd444b-2p7zz\" (UID: \"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.212840 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.212751 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:22.250086 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.250054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:22.326685 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.326279 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:22.354978 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.354945 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv"] Jul 31 20:03:22.356962 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:03:22.356921 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedaa52f_8b8e_462f_83e7_fcf362f38826.slice/crio-92c0a2a767d2df383ef344e9a76c5fe930fb16e9c4a523063df7c6662c63c13a WatchSource:0}: Error finding container 92c0a2a767d2df383ef344e9a76c5fe930fb16e9c4a523063df7c6662c63c13a: Status 404 returned error can't find the container with id 92c0a2a767d2df383ef344e9a76c5fe930fb16e9c4a523063df7c6662c63c13a Jul 31 20:03:22.401490 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.401442 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2p7zz"] Jul 31 20:03:22.403498 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:03:22.403459 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3916c_31ab_4d2e_9bd7_19ec0eab3033.slice/crio-3915d3ff62d258dbe1c12c37ea5ec85216ec8f8991d68313de810ecf636db355 WatchSource:0}: Error finding container 3915d3ff62d258dbe1c12c37ea5ec85216ec8f8991d68313de810ecf636db355: Status 404 returned error can't find the container with id 3915d3ff62d258dbe1c12c37ea5ec85216ec8f8991d68313de810ecf636db355 Jul 31 20:03:22.468984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.468956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c4944758f-r5l5m"] Jul 31 20:03:22.470717 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:03:22.470689 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9883ae5_2328_48fe_aa30_c3550ea96097.slice/crio-01aaf6b67e3e7f1dbb705d418391780dfd4337b0e3e83f90f468059509dabe4b WatchSource:0}: Error finding container 01aaf6b67e3e7f1dbb705d418391780dfd4337b0e3e83f90f468059509dabe4b: Status 404 returned error can't find the container with id 01aaf6b67e3e7f1dbb705d418391780dfd4337b0e3e83f90f468059509dabe4b Jul 31 20:03:22.612092 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.612054 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" event={"ID":"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033","Type":"ContainerStarted","Data":"3915d3ff62d258dbe1c12c37ea5ec85216ec8f8991d68313de810ecf636db355"} Jul 31 20:03:22.613290 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.613259 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" event={"ID":"eedaa52f-8b8e-462f-83e7-fcf362f38826","Type":"ContainerStarted","Data":"92c0a2a767d2df383ef344e9a76c5fe930fb16e9c4a523063df7c6662c63c13a"} Jul 31 20:03:22.614364 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:22.614339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" event={"ID":"a9883ae5-2328-48fe-aa30-c3550ea96097","Type":"ContainerStarted","Data":"01aaf6b67e3e7f1dbb705d418391780dfd4337b0e3e83f90f468059509dabe4b"} Jul 31 20:03:25.634544 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.634512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" event={"ID":"a2e3916c-31ab-4d2e-9bd7-19ec0eab3033","Type":"ContainerStarted","Data":"6e3a371992d675f20a99e2170257a4ebf19b7676be88a333e4f99fcf7ca5d00c"} Jul 31 20:03:25.635023 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.634710 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:25.636169 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.636143 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" event={"ID":"eedaa52f-8b8e-462f-83e7-fcf362f38826","Type":"ContainerStarted","Data":"aa53a6a919af5bc5f8a07a904b4579799e0fa522fdf9946a218853657e2ffc0c"} Jul 31 20:03:25.636287 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.636192 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:25.637595 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.637574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" event={"ID":"a9883ae5-2328-48fe-aa30-c3550ea96097","Type":"ContainerStarted","Data":"93161c41d1ecf5c5bbab4a464f8612e64d7fe1da4412a356d505fe0c738c865c"} Jul 31 20:03:25.637708 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.637693 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:25.648755 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.648718 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" podStartSLOduration=2.479769256 podStartE2EDuration="4.648707798s" totalImagesPullingTime="2.168938542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:03:25.647687431 +0000 UTC m=+534.989271059" watchObservedRunningTime="2026-07-31 20:03:25.648707798 +0000 UTC m=+534.990291426" Jul 31 20:03:25.664876 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.664829 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" podStartSLOduration=2.562055334 podStartE2EDuration="4.664816486s" totalImagesPullingTime="2.102761152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:03:25.662746877 +0000 UTC m=+535.004330544" watchObservedRunningTime="2026-07-31 20:03:25.664816486 +0000 UTC m=+535.006400146" Jul 31 20:03:25.677541 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:25.677501 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" podStartSLOduration=2.455248495 podStartE2EDuration="4.677489539s" totalImagesPullingTime="2.222241044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:03:25.676635497 +0000 UTC m=+535.018219126" watchObservedRunningTime="2026-07-31 20:03:25.677489539 +0000 UTC m=+535.019073166" Jul 31 20:03:31.647620 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:31.647589 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c4944758f-r5l5m" Jul 31 20:03:36.645161 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:36.645124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-b47wv" Jul 31 20:03:36.645594 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:36.645286 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-2p7zz" Jul 31 20:03:38.001078 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.001046 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6"] Jul 31 20:03:38.006025 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.006004 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.008522 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.008496 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 20:03:38.008644 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.008496 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 20:03:38.009417 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.009376 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wq5vw\"" Jul 31 20:03:38.009417 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.009410 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 31 20:03:38.014645 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.014624 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6"] Jul 31 20:03:38.158040 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.158176 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8bee38b-dddd-43cf-b959-5409f024f3b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.158176 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhphp\" (UniqueName: \"kubernetes.io/projected/d8bee38b-dddd-43cf-b959-5409f024f3b3-kube-api-access-nhphp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.158176 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158142 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.158289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.158289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.158217 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.258771 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.258694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8bee38b-dddd-43cf-b959-5409f024f3b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.258771 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.258729 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhphp\" (UniqueName: \"kubernetes.io/projected/d8bee38b-dddd-43cf-b959-5409f024f3b3-kube-api-access-nhphp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.258980 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.258866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.258980 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.258923 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.258980 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.258968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.259133 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.259039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.259277 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.259253 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.259377 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.259358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.259434 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.259408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.261077 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.261053 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8bee38b-dddd-43cf-b959-5409f024f3b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.261446 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.261429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8bee38b-dddd-43cf-b959-5409f024f3b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.266294 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.266273 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhphp\" (UniqueName: \"kubernetes.io/projected/d8bee38b-dddd-43cf-b959-5409f024f3b3-kube-api-access-nhphp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6\" (UID: \"d8bee38b-dddd-43cf-b959-5409f024f3b3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.318339 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.318289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:38.455539 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.455471 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6"] Jul 31 20:03:38.457779 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:03:38.457747 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bee38b_dddd_43cf_b959_5409f024f3b3.slice/crio-8e5358decdb4a7a50f1a89a435b7410e5a83ed32d8e8331532bf76d8547728a7 WatchSource:0}: Error finding container 8e5358decdb4a7a50f1a89a435b7410e5a83ed32d8e8331532bf76d8547728a7: Status 404 returned error can't find the container with id 8e5358decdb4a7a50f1a89a435b7410e5a83ed32d8e8331532bf76d8547728a7 Jul 31 20:03:38.707991 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:38.707954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" event={"ID":"d8bee38b-dddd-43cf-b959-5409f024f3b3","Type":"ContainerStarted","Data":"8e5358decdb4a7a50f1a89a435b7410e5a83ed32d8e8331532bf76d8547728a7"} Jul 31 20:03:39.714100 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:39.714059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" event={"ID":"d8bee38b-dddd-43cf-b959-5409f024f3b3","Type":"ContainerStarted","Data":"bfc1e79fa780c9c021ee8ab9332e07c75c95946de9123ed4d43ab121ad527021"} Jul 31 20:03:39.730141 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:39.730098 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" podStartSLOduration=1.631795237 podStartE2EDuration="2.730084916s" totalImagesPullingTime="1.098289679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:03:39.729000286 +0000 UTC m=+549.070583918" watchObservedRunningTime="2026-07-31 20:03:39.730084916 +0000 UTC m=+549.071668543" Jul 31 20:03:48.319451 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:48.319405 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:48.319937 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:48.319472 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:48.332852 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:48.332818 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:48.768955 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:48.768882 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6" Jul 31 20:03:52.301925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.301895 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg"] Jul 31 20:03:52.304937 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.304905 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.307148 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.307129 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 31 20:03:52.314818 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.314793 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg"] Jul 31 20:03:52.383933 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.383903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.384079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.383940 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.384079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.383969 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156b27d1-1e97-485e-834a-1a38526461a6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.384079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.383995 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.384079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.384028 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.384079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.384060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr74g\" (UniqueName: \"kubernetes.io/projected/156b27d1-1e97-485e-834a-1a38526461a6-kube-api-access-jr74g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484576 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484723 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484583 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr74g\" (UniqueName: \"kubernetes.io/projected/156b27d1-1e97-485e-834a-1a38526461a6-kube-api-access-jr74g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484723 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484723 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484723 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484699 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156b27d1-1e97-485e-834a-1a38526461a6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.484912 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.485004 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.484984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.485075 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.485053 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.485144 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.485125 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.486772 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.486749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156b27d1-1e97-485e-834a-1a38526461a6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.487162 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.487144 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156b27d1-1e97-485e-834a-1a38526461a6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.492055 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.492035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr74g\" (UniqueName: \"kubernetes.io/projected/156b27d1-1e97-485e-834a-1a38526461a6-kube-api-access-jr74g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-57cwg\" (UID: \"156b27d1-1e97-485e-834a-1a38526461a6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.616250 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.616218 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:03:52.753259 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.753233 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg"] Jul 31 20:03:52.755159 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:03:52.755129 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156b27d1_1e97_485e_834a_1a38526461a6.slice/crio-267abfa2882f0d89b324113797ac9683098e6952b434da095e863ff078ccde13 WatchSource:0}: Error finding container 267abfa2882f0d89b324113797ac9683098e6952b434da095e863ff078ccde13: Status 404 returned error can't find the container with id 267abfa2882f0d89b324113797ac9683098e6952b434da095e863ff078ccde13 Jul 31 20:03:52.778790 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:52.778765 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" event={"ID":"156b27d1-1e97-485e-834a-1a38526461a6","Type":"ContainerStarted","Data":"267abfa2882f0d89b324113797ac9683098e6952b434da095e863ff078ccde13"} Jul 31 20:03:53.785507 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:53.785468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" event={"ID":"156b27d1-1e97-485e-834a-1a38526461a6","Type":"ContainerStarted","Data":"193dacccae0019764ac5d7d49054c832417bc8fa8d0ca4310db26af499e3eb85"} Jul 31 20:03:53.809898 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:03:53.809847 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" podStartSLOduration=1.616183104 podStartE2EDuration="1.80982854s" totalImagesPullingTime="193.645436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:03:53.80884979 +0000 UTC m=+563.150433418" watchObservedRunningTime="2026-07-31 20:03:53.80982854 +0000 UTC m=+563.151412167" Jul 31 20:04:02.616368 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:02.616336 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:04:02.616753 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:02.616407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:04:02.629019 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:02.628990 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:04:02.840466 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:02.840439 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-57cwg" Jul 31 20:04:12.410031 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.409994 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj"] Jul 31 20:04:12.414285 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.414265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.416600 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.416581 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 31 20:04:12.421688 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.421664 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj"] Jul 31 20:04:12.442604 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442581 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.442714 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.442714 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442663 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.442792 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442714 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.442792 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.442792 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.442775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg54r\" (UniqueName: \"kubernetes.io/projected/846dfaf0-8e28-4ef5-803a-9b61d86b013a-kube-api-access-hg54r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.543805 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.543805 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg54r\" (UniqueName: \"kubernetes.io/projected/846dfaf0-8e28-4ef5-803a-9b61d86b013a-kube-api-access-hg54r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.543892 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.544235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544328 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.544246 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.544328 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.544289 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.546123 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.546096 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/846dfaf0-8e28-4ef5-803a-9b61d86b013a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.546269 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.546252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/846dfaf0-8e28-4ef5-803a-9b61d86b013a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.551456 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.551435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg54r\" (UniqueName: \"kubernetes.io/projected/846dfaf0-8e28-4ef5-803a-9b61d86b013a-kube-api-access-hg54r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hnxwj\" (UID: \"846dfaf0-8e28-4ef5-803a-9b61d86b013a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.726455 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.726375 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:12.853198 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.853174 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj"] Jul 31 20:04:12.854716 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:04:12.854678 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846dfaf0_8e28_4ef5_803a_9b61d86b013a.slice/crio-9fc9a03466ddbd1ce1b3a57fdb7599014ce5ff6ec1d96493e3021a86de9bfd51 WatchSource:0}: Error finding container 9fc9a03466ddbd1ce1b3a57fdb7599014ce5ff6ec1d96493e3021a86de9bfd51: Status 404 returned error can't find the container with id 9fc9a03466ddbd1ce1b3a57fdb7599014ce5ff6ec1d96493e3021a86de9bfd51 Jul 31 20:04:12.879321 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:12.879284 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" event={"ID":"846dfaf0-8e28-4ef5-803a-9b61d86b013a","Type":"ContainerStarted","Data":"9fc9a03466ddbd1ce1b3a57fdb7599014ce5ff6ec1d96493e3021a86de9bfd51"} Jul 31 20:04:13.886009 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:13.885977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" event={"ID":"846dfaf0-8e28-4ef5-803a-9b61d86b013a","Type":"ContainerStarted","Data":"5c5eccf605d5e9e713b6035480bb03d0d58327982856d3321d7b2778c95ab347"} Jul 31 20:04:13.902659 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:13.902613 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" podStartSLOduration=1.718605196 podStartE2EDuration="1.902595601s" totalImagesPullingTime="183.990405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:04:13.901457624 +0000 UTC m=+583.243041250" watchObservedRunningTime="2026-07-31 20:04:13.902595601 +0000 UTC m=+583.244179227" Jul 31 20:04:22.726529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:22.726498 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:22.726920 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:22.726574 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:22.738470 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:22.738446 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:22.943973 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:22.943940 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hnxwj" Jul 31 20:04:31.106097 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:31.106066 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:04:31.116243 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:04:31.116224 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:06:21.964865 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:21.964824 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:06:21.965405 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:21.965152 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-777c587c75-wzp6s" podUID="c7744847-4a64-41b7-837b-33d8377fb500" containerName="manager" containerID="cri-o://d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9" gracePeriod=30 Jul 31 20:06:22.208971 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.208945 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:06:22.314252 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.314228 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") pod \"c7744847-4a64-41b7-837b-33d8377fb500\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " Jul 31 20:06:22.314445 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.314268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8dnj\" (UniqueName: \"kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj\") pod \"c7744847-4a64-41b7-837b-33d8377fb500\" (UID: \"c7744847-4a64-41b7-837b-33d8377fb500\") " Jul 31 20:06:22.316363 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.316334 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj" (OuterVolumeSpecName: "kube-api-access-l8dnj") pod "c7744847-4a64-41b7-837b-33d8377fb500" (UID: "c7744847-4a64-41b7-837b-33d8377fb500"). InnerVolumeSpecName "kube-api-access-l8dnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:06:22.316480 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.316340 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c7744847-4a64-41b7-837b-33d8377fb500" (UID: "c7744847-4a64-41b7-837b-33d8377fb500"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:06:22.415245 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.415218 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7744847-4a64-41b7-837b-33d8377fb500-webhook-cert\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:06:22.415245 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.415245 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dnj\" (UniqueName: \"kubernetes.io/projected/c7744847-4a64-41b7-837b-33d8377fb500-kube-api-access-l8dnj\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:06:22.538903 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.538865 2556 generic.go:356] "Generic (PLEG): container finished" podID="c7744847-4a64-41b7-837b-33d8377fb500" containerID="d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9" exitCode=0 Jul 31 20:06:22.539068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.538908 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-wzp6s" Jul 31 20:06:22.539068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.538908 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-wzp6s" event={"ID":"c7744847-4a64-41b7-837b-33d8377fb500","Type":"ContainerDied","Data":"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9"} Jul 31 20:06:22.539068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.539041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-wzp6s" event={"ID":"c7744847-4a64-41b7-837b-33d8377fb500","Type":"ContainerDied","Data":"c1baecc17021d7756a74e022d05b6ee5df984754335fea0c1aef9132b9113666"} Jul 31 20:06:22.539068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.539067 2556 scope.go:117] "RemoveContainer" containerID="d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9" Jul 31 20:06:22.547814 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.547794 2556 scope.go:117] "RemoveContainer" containerID="d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9" Jul 31 20:06:22.548035 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:06:22.548017 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9\": container with ID starting with d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9 not found: ID does not exist" containerID="d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9" Jul 31 20:06:22.548079 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.548043 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9"} err="failed to get container status \"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9\": rpc error: code = NotFound desc = could not find container \"d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9\": container with ID starting with d6bac0b6d7a12066132394f530a0c5cef387a0a2616c719e75997bac79a723c9 not found: ID does not exist" Jul 31 20:06:22.558702 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.558678 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:06:22.566680 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:22.566633 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-777c587c75-wzp6s"] Jul 31 20:06:23.166226 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:06:23.166183 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7744847-4a64-41b7-837b-33d8377fb500" path="/var/lib/kubelet/pods/c7744847-4a64-41b7-837b-33d8377fb500/volumes" Jul 31 20:07:23.393142 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.393093 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:23.393574 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.393447 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7744847-4a64-41b7-837b-33d8377fb500" containerName="manager" Jul 31 20:07:23.393574 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.393459 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7744847-4a64-41b7-837b-33d8377fb500" containerName="manager" Jul 31 20:07:23.393574 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.393517 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7744847-4a64-41b7-837b-33d8377fb500" containerName="manager" Jul 31 20:07:23.396383 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.396366 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.399606 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.399583 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 20:07:23.399713 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.399629 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gg4jb\"" Jul 31 20:07:23.399713 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.399657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 20:07:23.405592 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.405571 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:23.502081 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.502054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbk9\" (UniqueName: \"kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.502210 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.502118 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.603095 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.603063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.603265 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.603112 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbk9\" (UniqueName: \"kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.605505 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.605481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.610668 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.610647 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbk9\" (UniqueName: \"kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9\") pod \"maas-controller-777c587c75-xc7j6\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.707071 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.706993 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:23.835352 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.835306 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:23.836577 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:07:23.836552 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e01a72_9bd9_4b04_9168_30b4be7e467b.slice/crio-439004f45208d6163d4f90d4d063e2b915e9163cd2e1bf060655f84d8a63c163 WatchSource:0}: Error finding container 439004f45208d6163d4f90d4d063e2b915e9163cd2e1bf060655f84d8a63c163: Status 404 returned error can't find the container with id 439004f45208d6163d4f90d4d063e2b915e9163cd2e1bf060655f84d8a63c163 Jul 31 20:07:23.837847 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:23.837832 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:07:24.841115 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:24.841078 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-xc7j6" event={"ID":"85e01a72-9bd9-4b04-9168-30b4be7e467b","Type":"ContainerStarted","Data":"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35"} Jul 31 20:07:24.841527 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:24.841133 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-xc7j6" event={"ID":"85e01a72-9bd9-4b04-9168-30b4be7e467b","Type":"ContainerStarted","Data":"439004f45208d6163d4f90d4d063e2b915e9163cd2e1bf060655f84d8a63c163"} Jul 31 20:07:24.841527 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:24.841153 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:24.857969 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:24.857913 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777c587c75-xc7j6" podStartSLOduration=1.070808354 podStartE2EDuration="1.857900143s" totalImagesPullingTime="787.091789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:07:24.856449501 +0000 UTC m=+774.198033125" watchObservedRunningTime="2026-07-31 20:07:24.857900143 +0000 UTC m=+774.199483768" Jul 31 20:07:35.847747 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:35.847717 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:40.519645 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.519613 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:40.520074 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.519796 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-777c587c75-xc7j6" podUID="85e01a72-9bd9-4b04-9168-30b4be7e467b" containerName="manager" containerID="cri-o://88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35" gracePeriod=30 Jul 31 20:07:40.758730 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.758707 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:40.850109 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.850082 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tbk9\" (UniqueName: \"kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9\") pod \"85e01a72-9bd9-4b04-9168-30b4be7e467b\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " Jul 31 20:07:40.850275 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.850136 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert\") pod \"85e01a72-9bd9-4b04-9168-30b4be7e467b\" (UID: \"85e01a72-9bd9-4b04-9168-30b4be7e467b\") " Jul 31 20:07:40.851998 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.851965 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9" (OuterVolumeSpecName: "kube-api-access-8tbk9") pod "85e01a72-9bd9-4b04-9168-30b4be7e467b" (UID: "85e01a72-9bd9-4b04-9168-30b4be7e467b"). InnerVolumeSpecName "kube-api-access-8tbk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:07:40.852101 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.852088 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "85e01a72-9bd9-4b04-9168-30b4be7e467b" (UID: "85e01a72-9bd9-4b04-9168-30b4be7e467b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:07:40.921923 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.921890 2556 generic.go:356] "Generic (PLEG): container finished" podID="85e01a72-9bd9-4b04-9168-30b4be7e467b" containerID="88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35" exitCode=0 Jul 31 20:07:40.922054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.921936 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-xc7j6" Jul 31 20:07:40.922054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.921951 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-xc7j6" event={"ID":"85e01a72-9bd9-4b04-9168-30b4be7e467b","Type":"ContainerDied","Data":"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35"} Jul 31 20:07:40.922054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.921998 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-xc7j6" event={"ID":"85e01a72-9bd9-4b04-9168-30b4be7e467b","Type":"ContainerDied","Data":"439004f45208d6163d4f90d4d063e2b915e9163cd2e1bf060655f84d8a63c163"} Jul 31 20:07:40.922054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.922021 2556 scope.go:117] "RemoveContainer" containerID="88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35" Jul 31 20:07:40.930696 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.930679 2556 scope.go:117] "RemoveContainer" containerID="88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35" Jul 31 20:07:40.930949 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:07:40.930932 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35\": container with ID starting with 88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35 not found: ID does not exist" containerID="88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35" Jul 31 20:07:40.930998 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.930958 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35"} err="failed to get container status \"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35\": rpc error: code = NotFound desc = could not find container \"88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35\": container with ID starting with 88fc3392a01100467675062da8e9d1721eaffcc0b64a0b647fd1bacbb5894b35 not found: ID does not exist" Jul 31 20:07:40.940731 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.940706 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:40.943485 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.943466 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-777c587c75-xc7j6"] Jul 31 20:07:40.951708 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.951689 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbk9\" (UniqueName: \"kubernetes.io/projected/85e01a72-9bd9-4b04-9168-30b4be7e467b-kube-api-access-8tbk9\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:07:40.951761 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:40.951712 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e01a72-9bd9-4b04-9168-30b4be7e467b-webhook-cert\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:07:41.166136 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.166063 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85e01a72-9bd9-4b04-9168-30b4be7e467b" path="/var/lib/kubelet/pods/85e01a72-9bd9-4b04-9168-30b4be7e467b/volumes" Jul 31 20:07:41.756276 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.756243 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777c587c75-l75cv"] Jul 31 20:07:41.756657 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.756615 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e01a72-9bd9-4b04-9168-30b4be7e467b" containerName="manager" Jul 31 20:07:41.756657 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.756628 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e01a72-9bd9-4b04-9168-30b4be7e467b" containerName="manager" Jul 31 20:07:41.756736 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.756688 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e01a72-9bd9-4b04-9168-30b4be7e467b" containerName="manager" Jul 31 20:07:41.760872 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.760855 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.763011 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.762989 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 20:07:41.764221 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.764196 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gg4jb\"" Jul 31 20:07:41.764353 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.764230 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 20:07:41.767276 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.767251 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-l75cv"] Jul 31 20:07:41.860220 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.860189 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd26w\" (UniqueName: \"kubernetes.io/projected/37769134-9c91-44af-b193-c5b1d046b730-kube-api-access-jd26w\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.860407 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.860259 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37769134-9c91-44af-b193-c5b1d046b730-webhook-cert\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.960946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.960919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd26w\" (UniqueName: \"kubernetes.io/projected/37769134-9c91-44af-b193-c5b1d046b730-kube-api-access-jd26w\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.961120 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.960984 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37769134-9c91-44af-b193-c5b1d046b730-webhook-cert\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.963308 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.963287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37769134-9c91-44af-b193-c5b1d046b730-webhook-cert\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:41.968670 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:41.968641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd26w\" (UniqueName: \"kubernetes.io/projected/37769134-9c91-44af-b193-c5b1d046b730-kube-api-access-jd26w\") pod \"maas-controller-777c587c75-l75cv\" (UID: \"37769134-9c91-44af-b193-c5b1d046b730\") " pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:42.072161 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:42.072122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:42.199481 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:42.199445 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777c587c75-l75cv"] Jul 31 20:07:42.200852 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:07:42.200826 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37769134_9c91_44af_b193_c5b1d046b730.slice/crio-e6434ceb52f31f801b0c98053b2e344ed7a15827ea64f6dbbebe4bd449d113c0 WatchSource:0}: Error finding container e6434ceb52f31f801b0c98053b2e344ed7a15827ea64f6dbbebe4bd449d113c0: Status 404 returned error can't find the container with id e6434ceb52f31f801b0c98053b2e344ed7a15827ea64f6dbbebe4bd449d113c0 Jul 31 20:07:42.933105 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:42.933063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-l75cv" event={"ID":"37769134-9c91-44af-b193-c5b1d046b730","Type":"ContainerStarted","Data":"e6434ceb52f31f801b0c98053b2e344ed7a15827ea64f6dbbebe4bd449d113c0"} Jul 31 20:07:43.940026 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:43.939988 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777c587c75-l75cv" event={"ID":"37769134-9c91-44af-b193-c5b1d046b730","Type":"ContainerStarted","Data":"f922b60bd5bc1df6b124d7063ca586581a47bf52bb334a90284b4fb7d64f3f29"} Jul 31 20:07:43.940433 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:43.940110 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:07:43.956997 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:43.956947 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777c587c75-l75cv" podStartSLOduration=1.6792442429999999 podStartE2EDuration="2.956932575s" totalImagesPullingTime="1.277688332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:07:43.954472668 +0000 UTC m=+793.296056294" watchObservedRunningTime="2026-07-31 20:07:43.956932575 +0000 UTC m=+793.298516201" Jul 31 20:07:54.947270 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:07:54.947200 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777c587c75-l75cv" Jul 31 20:09:31.144521 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:09:31.144446 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:09:31.151758 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:09:31.151737 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:14:14.545802 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.545763 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:14:14.546234 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.545958 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" podUID="86073eaf-f62a-4376-92c2-e01644d1645e" containerName="manager" containerID="cri-o://4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd" gracePeriod=10 Jul 31 20:14:14.805331 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.805292 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:14:14.869569 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.869530 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume\") pod \"86073eaf-f62a-4376-92c2-e01644d1645e\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " Jul 31 20:14:14.869725 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.869657 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blr4z\" (UniqueName: \"kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z\") pod \"86073eaf-f62a-4376-92c2-e01644d1645e\" (UID: \"86073eaf-f62a-4376-92c2-e01644d1645e\") " Jul 31 20:14:14.869910 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.869885 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "86073eaf-f62a-4376-92c2-e01644d1645e" (UID: "86073eaf-f62a-4376-92c2-e01644d1645e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:14:14.871579 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.871543 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z" (OuterVolumeSpecName: "kube-api-access-blr4z") pod "86073eaf-f62a-4376-92c2-e01644d1645e" (UID: "86073eaf-f62a-4376-92c2-e01644d1645e"). InnerVolumeSpecName "kube-api-access-blr4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:14:14.930400 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.930358 2556 generic.go:356] "Generic (PLEG): container finished" podID="86073eaf-f62a-4376-92c2-e01644d1645e" containerID="4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd" exitCode=0 Jul 31 20:14:14.930551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.930416 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" Jul 31 20:14:14.930551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.930458 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" event={"ID":"86073eaf-f62a-4376-92c2-e01644d1645e","Type":"ContainerDied","Data":"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd"} Jul 31 20:14:14.930551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.930520 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb" event={"ID":"86073eaf-f62a-4376-92c2-e01644d1645e","Type":"ContainerDied","Data":"2a6840aa0d96911c22b3ed658346096a0a73c13b1f08bff2fd4c71f8894548c7"} Jul 31 20:14:14.930551 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.930544 2556 scope.go:117] "RemoveContainer" containerID="4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd" Jul 31 20:14:14.940632 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.940615 2556 scope.go:117] "RemoveContainer" containerID="4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd" Jul 31 20:14:14.940884 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:14:14.940866 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd\": container with ID starting with 4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd not found: ID does not exist" containerID="4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd" Jul 31 20:14:14.940932 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.940894 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd"} err="failed to get container status \"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd\": rpc error: code = NotFound desc = could not find container \"4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd\": container with ID starting with 4e6211b7ca7aa32a449124a7d9d84f72c60a5cfe80e2c18c4c2c2a633e9089dd not found: ID does not exist" Jul 31 20:14:14.952044 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.952021 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:14:14.955672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.955652 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hrmhb"] Jul 31 20:14:14.970811 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.970789 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blr4z\" (UniqueName: \"kubernetes.io/projected/86073eaf-f62a-4376-92c2-e01644d1645e-kube-api-access-blr4z\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:14:14.970811 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:14.970809 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86073eaf-f62a-4376-92c2-e01644d1645e-extensions-socket-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:14:15.166495 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:15.166423 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86073eaf-f62a-4376-92c2-e01644d1645e" path="/var/lib/kubelet/pods/86073eaf-f62a-4376-92c2-e01644d1645e/volumes" Jul 31 20:14:31.182438 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:31.182406 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:14:31.185703 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:14:31.185681 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:15:00.141639 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.141600 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2"] Jul 31 20:15:00.142226 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.142115 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86073eaf-f62a-4376-92c2-e01644d1645e" containerName="manager" Jul 31 20:15:00.142226 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.142137 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="86073eaf-f62a-4376-92c2-e01644d1645e" containerName="manager" Jul 31 20:15:00.142388 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.142237 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="86073eaf-f62a-4376-92c2-e01644d1645e" containerName="manager" Jul 31 20:15:00.145209 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.145187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" Jul 31 20:15:00.147707 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.147685 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkrj7\"" Jul 31 20:15:00.153085 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.153063 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2"] Jul 31 20:15:00.156871 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.156850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" Jul 31 20:15:00.288653 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.288605 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2"] Jul 31 20:15:00.293944 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:15:00.293918 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b4d9b3_0f25_433b_a87e_3108852a1170.slice/crio-9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6 WatchSource:0}: Error finding container 9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6: Status 404 returned error can't find the container with id 9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6 Jul 31 20:15:00.295708 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:00.295689 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:15:01.157753 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:01.157716 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" event={"ID":"c6b4d9b3-0f25-433b-a87e-3108852a1170","Type":"ContainerStarted","Data":"9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6"} Jul 31 20:15:04.176161 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:04.176127 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" event={"ID":"c6b4d9b3-0f25-433b-a87e-3108852a1170","Type":"ContainerStarted","Data":"ac4284b7c5473e1af4010e4e74a77fd9872473c86ca48fd4e291e17c326c2d41"} Jul 31 20:15:04.190330 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:04.190263 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" podStartSLOduration=1.129701472 podStartE2EDuration="4.190249062s" totalImagesPullingTime="3.06054759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:15:04.188739873 +0000 UTC m=+1233.530323511" watchObservedRunningTime="2026-07-31 20:15:04.190249062 +0000 UTC m=+1233.531832689" Jul 31 20:15:05.181630 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:05.181594 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6b4d9b3-0f25-433b-a87e-3108852a1170" containerID="ac4284b7c5473e1af4010e4e74a77fd9872473c86ca48fd4e291e17c326c2d41" exitCode=0 Jul 31 20:15:05.182008 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:05.181677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" event={"ID":"c6b4d9b3-0f25-433b-a87e-3108852a1170","Type":"ContainerDied","Data":"ac4284b7c5473e1af4010e4e74a77fd9872473c86ca48fd4e291e17c326c2d41"} Jul 31 20:15:06.323964 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:06.323932 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" Jul 31 20:15:07.192835 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:07.192803 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" event={"ID":"c6b4d9b3-0f25-433b-a87e-3108852a1170","Type":"ContainerDied","Data":"9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6"} Jul 31 20:15:07.192835 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:07.192829 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758815-whvk2" Jul 31 20:15:07.193033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:07.192849 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9297be6b31e2c44a9dd273eb66c14e61c9b5c4e35feb44776f5b28fe70efd6e6" Jul 31 20:15:15.266033 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.265995 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d"] Jul 31 20:15:15.266609 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.266588 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b4d9b3-0f25-433b-a87e-3108852a1170" containerName="cleanup" Jul 31 20:15:15.266687 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.266611 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b4d9b3-0f25-433b-a87e-3108852a1170" containerName="cleanup" Jul 31 20:15:15.266739 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.266711 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b4d9b3-0f25-433b-a87e-3108852a1170" containerName="cleanup" Jul 31 20:15:15.269746 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.269727 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.272123 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.272104 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j2f22\"" Jul 31 20:15:15.280659 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.280633 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d"] Jul 31 20:15:15.394036 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.393997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkjd\" (UniqueName: \"kubernetes.io/projected/2fe4b7c5-38f7-4154-870a-aa040c467a6e-kube-api-access-hfkjd\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.394233 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.394187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fe4b7c5-38f7-4154-870a-aa040c467a6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.494627 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.494594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fe4b7c5-38f7-4154-870a-aa040c467a6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.494773 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.494645 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkjd\" (UniqueName: \"kubernetes.io/projected/2fe4b7c5-38f7-4154-870a-aa040c467a6e-kube-api-access-hfkjd\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.494984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.494964 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fe4b7c5-38f7-4154-870a-aa040c467a6e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.504005 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.503974 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkjd\" (UniqueName: \"kubernetes.io/projected/2fe4b7c5-38f7-4154-870a-aa040c467a6e-kube-api-access-hfkjd\") pod \"kuadrant-operator-controller-manager-68cf589589-hn97d\" (UID: \"2fe4b7c5-38f7-4154-870a-aa040c467a6e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.580155 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.580111 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:15.718185 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:15.718156 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d"] Jul 31 20:15:15.719035 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:15:15.718999 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe4b7c5_38f7_4154_870a_aa040c467a6e.slice/crio-1cb948d003ef993f19aa428c07a3d84858adf8ecf809da933c8fda035a104463 WatchSource:0}: Error finding container 1cb948d003ef993f19aa428c07a3d84858adf8ecf809da933c8fda035a104463: Status 404 returned error can't find the container with id 1cb948d003ef993f19aa428c07a3d84858adf8ecf809da933c8fda035a104463 Jul 31 20:15:16.239580 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:16.239545 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" event={"ID":"2fe4b7c5-38f7-4154-870a-aa040c467a6e","Type":"ContainerStarted","Data":"2bfaa1c5ddcd83015f968538fccb9b4d9fa5114380332c4b38a5baa15f4f3cb1"} Jul 31 20:15:16.239760 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:16.239593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" event={"ID":"2fe4b7c5-38f7-4154-870a-aa040c467a6e","Type":"ContainerStarted","Data":"1cb948d003ef993f19aa428c07a3d84858adf8ecf809da933c8fda035a104463"} Jul 31 20:15:16.239760 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:16.239671 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:15:16.264642 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:16.264590 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" podStartSLOduration=1.264574426 podStartE2EDuration="1.264574426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:15:16.262895103 +0000 UTC m=+1245.604478732" watchObservedRunningTime="2026-07-31 20:15:16.264574426 +0000 UTC m=+1245.606158056" Jul 31 20:15:27.246851 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:15:27.246775 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-hn97d" Jul 31 20:18:12.913120 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:12.913088 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:18:12.916450 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:12.916433 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:12.918584 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:12.918568 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b066245f-kserve-self-signed-certs\"" Jul 31 20:18:12.933023 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:12.932999 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:18:13.025887 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.025854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.026068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.025892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.026068 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.025928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.026384 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.026069 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.026384 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.026109 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.026384 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.026159 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42z6\" (UniqueName: \"kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.126949 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.126923 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.126963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.126981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42z6\" (UniqueName: \"kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127071 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127550 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127525 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127651 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127551 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.127710 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.127664 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.129341 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.129306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.129582 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.129566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.134602 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.134578 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42z6\" (UniqueName: \"kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6\") pod \"e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.224811 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.224711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:13.364441 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:13.362456 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:18:13.366786 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:18:13.366758 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e6a33cd_20a4_4736_bca8_fb6d881a12d4.slice/crio-3e4637c8a1f67c98c7205ae3fc6408df7a37e983a5757371ef85cbd2fd13bbaa WatchSource:0}: Error finding container 3e4637c8a1f67c98c7205ae3fc6408df7a37e983a5757371ef85cbd2fd13bbaa: Status 404 returned error can't find the container with id 3e4637c8a1f67c98c7205ae3fc6408df7a37e983a5757371ef85cbd2fd13bbaa Jul 31 20:18:14.129620 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:14.129575 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" event={"ID":"8e6a33cd-20a4-4736-bca8-fb6d881a12d4","Type":"ContainerStarted","Data":"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986"} Jul 31 20:18:14.129984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:14.129633 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" event={"ID":"8e6a33cd-20a4-4736-bca8-fb6d881a12d4","Type":"ContainerStarted","Data":"3e4637c8a1f67c98c7205ae3fc6408df7a37e983a5757371ef85cbd2fd13bbaa"} Jul 31 20:18:14.146323 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:14.146270 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" podStartSLOduration=2.146256443 podStartE2EDuration="2.146256443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:18:14.145055052 +0000 UTC m=+1423.486638678" watchObservedRunningTime="2026-07-31 20:18:14.146256443 +0000 UTC m=+1423.487840151" Jul 31 20:18:23.225291 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:23.225185 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:23.225291 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:23.225256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:23.237529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:23.237502 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:18:24.193372 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:18:24.193330 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:19:13.557490 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:13.557446 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:19:28.577226 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.577180 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" podUID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" containerName="main" containerID="cri-o://99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986" gracePeriod=45 Jul 31 20:19:28.819808 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.819785 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:19:28.874596 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874533 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.874596 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874567 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.874797 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874612 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.874797 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874654 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x42z6\" (UniqueName: \"kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.874797 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874703 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.874797 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874754 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir\") pod \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\" (UID: \"8e6a33cd-20a4-4736-bca8-fb6d881a12d4\") " Jul 31 20:19:28.875059 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874751 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:19:28.875059 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874877 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home" (OuterVolumeSpecName: "home") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:19:28.875059 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.874957 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:19:28.875156 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.875057 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tmp-dir\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:28.875156 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.875074 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-model-cache\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:28.875156 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.875082 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-home\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:28.876819 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.876796 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm" (OuterVolumeSpecName: "dshm") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:19:28.876907 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.876824 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:19:28.876945 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.876907 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6" (OuterVolumeSpecName: "kube-api-access-x42z6") pod "8e6a33cd-20a4-4736-bca8-fb6d881a12d4" (UID: "8e6a33cd-20a4-4736-bca8-fb6d881a12d4"). InnerVolumeSpecName "kube-api-access-x42z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:19:28.975528 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.975494 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-dshm\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:28.975528 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.975526 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42z6\" (UniqueName: \"kubernetes.io/projected/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-kube-api-access-x42z6\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:28.975528 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:28.975536 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a33cd-20a4-4736-bca8-fb6d881a12d4-tls-certs\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:19:29.525105 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.525020 2556 generic.go:356] "Generic (PLEG): container finished" podID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" containerID="99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986" exitCode=0 Jul 31 20:19:29.525105 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.525060 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" Jul 31 20:19:29.525283 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.525109 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" event={"ID":"8e6a33cd-20a4-4736-bca8-fb6d881a12d4","Type":"ContainerDied","Data":"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986"} Jul 31 20:19:29.525283 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.525148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm" event={"ID":"8e6a33cd-20a4-4736-bca8-fb6d881a12d4","Type":"ContainerDied","Data":"3e4637c8a1f67c98c7205ae3fc6408df7a37e983a5757371ef85cbd2fd13bbaa"} Jul 31 20:19:29.525283 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.525168 2556 scope.go:117] "RemoveContainer" containerID="99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986" Jul 31 20:19:29.533481 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.533461 2556 scope.go:117] "RemoveContainer" containerID="99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986" Jul 31 20:19:29.533736 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:19:29.533715 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986\": container with ID starting with 99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986 not found: ID does not exist" containerID="99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986" Jul 31 20:19:29.533773 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.533748 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986"} err="failed to get container status \"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986\": rpc error: code = NotFound desc = could not find container \"99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986\": container with ID starting with 99fb7a2efebfc054c4651557470a83a2d0d2b39d766e3004450b74f334f45986 not found: ID does not exist" Jul 31 20:19:29.541846 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.541822 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:19:29.549305 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:29.549041 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b066245f-kserve-5765c75f59-w65bm"] Jul 31 20:19:31.167037 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:31.167005 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" path="/var/lib/kubelet/pods/8e6a33cd-20a4-4736-bca8-fb6d881a12d4/volumes" Jul 31 20:19:31.217946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:31.217926 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:19:31.219364 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:19:31.219340 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:24:31.256966 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:24:31.256940 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:24:31.257670 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:24:31.257646 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:29:31.291501 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:29:31.291473 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:29:31.294057 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:29:31.294034 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:33:23.783464 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.783382 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:33:23.783925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.783751 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" containerName="main" Jul 31 20:33:23.783925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.783762 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" containerName="main" Jul 31 20:33:23.783925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.783815 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6a33cd-20a4-4736-bca8-fb6d881a12d4" containerName="main" Jul 31 20:33:23.786872 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.786852 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.789071 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.789049 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1fa6acc8-serving-cert\"" Jul 31 20:33:23.795904 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.795876 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:33:23.862099 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.862049 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.862283 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.862115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.862283 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.862239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5cb\" (UniqueName: \"kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.963422 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.963383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.963605 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.963443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.963605 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.963544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5cb\" (UniqueName: \"kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.963701 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:33:23.963595 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-serving-cert: secret "maas-api-e2e-ait-1fa6acc8-serving-cert" not found Jul 31 20:33:23.963701 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:33:23.963688 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls podName:e2cb357f-6b83-4378-bb28-12800e77209a nodeName:}" failed. No retries permitted until 2026-07-31 20:33:24.463664948 +0000 UTC m=+2333.805248576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls") pod "maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" (UID: "e2cb357f-6b83-4378-bb28-12800e77209a") : secret "maas-api-e2e-ait-1fa6acc8-serving-cert" not found Jul 31 20:33:23.964092 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.964070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:23.974028 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:23.973997 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5cb\" (UniqueName: \"kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:24.401192 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.401156 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:33:24.404737 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.404720 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.407268 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.407242 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1fa6acc8\"" Jul 31 20:33:24.407268 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.407242 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1fa6acc8-dockercfg-7vfwl\"" Jul 31 20:33:24.412434 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.412407 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:33:24.428261 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.428233 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:33:24.432281 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.432262 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.441578 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.441555 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:33:24.469348 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.469292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.469529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.469365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:24.469529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.469411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.469529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.469450 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m97\" (UniqueName: \"kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.469529 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.469492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqkf\" (UniqueName: \"kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.471743 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.471722 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") pod \"maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:24.570616 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.570572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.570847 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.570644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m97\" (UniqueName: \"kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.570847 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.570690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqkf\" (UniqueName: \"kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.570847 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.570740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.571348 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.571300 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.571485 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.571465 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.579356 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.579328 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m97\" (UniqueName: \"kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97\") pod \"payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.579457 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.579408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqkf\" (UniqueName: \"kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf\") pod \"payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.699588 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.699489 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:24.717636 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.717610 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:24.743747 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.743713 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:24.875522 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.875485 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:33:24.877759 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:33:24.877714 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cb357f_6b83_4378_bb28_12800e77209a.slice/crio-2f00cf4348b46e5146423e101cc5dbb03d7e4f858059cea793eebcd0ac24a4e7 WatchSource:0}: Error finding container 2f00cf4348b46e5146423e101cc5dbb03d7e4f858059cea793eebcd0ac24a4e7: Status 404 returned error can't find the container with id 2f00cf4348b46e5146423e101cc5dbb03d7e4f858059cea793eebcd0ac24a4e7 Jul 31 20:33:24.879713 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.879660 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:33:24.908392 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.908362 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:33:24.911057 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:33:24.911012 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cbeb13_aeb6_48a7_a272_4bc594475490.slice/crio-a68e47c0f3629855181b0a177a6c026527575091e5504cb54376588f277d460d WatchSource:0}: Error finding container a68e47c0f3629855181b0a177a6c026527575091e5504cb54376588f277d460d: Status 404 returned error can't find the container with id a68e47c0f3629855181b0a177a6c026527575091e5504cb54376588f277d460d Jul 31 20:33:24.932258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:24.932202 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:33:24.934419 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:33:24.934390 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a48b1f9_9441_4f3b_96f6_7124c2b0ed79.slice/crio-1a8f99fc464a1cf937ce7fde2daad3b0adfdf5428873c085c2cac406b26ef4a9 WatchSource:0}: Error finding container 1a8f99fc464a1cf937ce7fde2daad3b0adfdf5428873c085c2cac406b26ef4a9: Status 404 returned error can't find the container with id 1a8f99fc464a1cf937ce7fde2daad3b0adfdf5428873c085c2cac406b26ef4a9 Jul 31 20:33:25.683127 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.682918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" event={"ID":"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79","Type":"ContainerStarted","Data":"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3"} Jul 31 20:33:25.683127 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.683028 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" event={"ID":"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79","Type":"ContainerStarted","Data":"1a8f99fc464a1cf937ce7fde2daad3b0adfdf5428873c085c2cac406b26ef4a9"} Jul 31 20:33:25.683127 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.683066 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:25.684745 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.684712 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" event={"ID":"19cbeb13-aeb6-48a7-a272-4bc594475490","Type":"ContainerStarted","Data":"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f"} Jul 31 20:33:25.684896 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.684770 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" event={"ID":"19cbeb13-aeb6-48a7-a272-4bc594475490","Type":"ContainerStarted","Data":"a68e47c0f3629855181b0a177a6c026527575091e5504cb54376588f277d460d"} Jul 31 20:33:25.684896 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.684798 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:33:25.686565 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.686541 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" event={"ID":"e2cb357f-6b83-4378-bb28-12800e77209a","Type":"ContainerStarted","Data":"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b"} Jul 31 20:33:25.686693 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.686587 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" event={"ID":"e2cb357f-6b83-4378-bb28-12800e77209a","Type":"ContainerStarted","Data":"2f00cf4348b46e5146423e101cc5dbb03d7e4f858059cea793eebcd0ac24a4e7"} Jul 31 20:33:25.686693 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.686615 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:25.699717 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.699674 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" podStartSLOduration=1.699661738 podStartE2EDuration="1.699661738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:33:25.697989484 +0000 UTC m=+2335.039573111" watchObservedRunningTime="2026-07-31 20:33:25.699661738 +0000 UTC m=+2335.041245421" Jul 31 20:33:25.712220 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.712184 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" podStartSLOduration=1.71216626 podStartE2EDuration="1.71216626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:33:25.711876613 +0000 UTC m=+2335.053460242" watchObservedRunningTime="2026-07-31 20:33:25.71216626 +0000 UTC m=+2335.053749882" Jul 31 20:33:25.728826 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:25.728788 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" podStartSLOduration=2.19802124 podStartE2EDuration="2.728775803s" totalImagesPullingTime="530.754563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:33:25.726549648 +0000 UTC m=+2335.068133296" watchObservedRunningTime="2026-07-31 20:33:25.728775803 +0000 UTC m=+2335.070359468" Jul 31 20:33:31.697477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:31.697445 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:33:36.694526 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:36.694493 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:33:36.694991 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:33:36.694864 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:34:31.324017 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:34:31.323985 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:34:31.328472 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:34:31.328451 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:36:36.810495 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.810436 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:36:36.811021 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.810690 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" podUID="e2cb357f-6b83-4378-bb28-12800e77209a" containerName="maas-api" containerID="cri-o://bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b" gracePeriod=30 Jul 31 20:36:36.869392 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.869360 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:36:36.869795 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.869632 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" podUID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" containerName="payload-processing" containerID="cri-o://85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3" gracePeriod=30 Jul 31 20:36:36.875800 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.875772 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:36:36.876037 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:36.875987 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" podUID="19cbeb13-aeb6-48a7-a272-4bc594475490" containerName="payload-pre-processing" containerID="cri-o://696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f" gracePeriod=30 Jul 31 20:36:37.128647 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.128624 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:36:37.166257 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.166231 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:36:37.189088 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.189061 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:36:37.228928 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.228896 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume\") pod \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " Jul 31 20:36:37.229107 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.228942 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89m97\" (UniqueName: \"kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97\") pod \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\" (UID: \"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79\") " Jul 31 20:36:37.229289 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.229267 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" (UID: "6a48b1f9-9441-4f3b-96f6-7124c2b0ed79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:36:37.231016 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.230996 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97" (OuterVolumeSpecName: "kube-api-access-89m97") pod "6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" (UID: "6a48b1f9-9441-4f3b-96f6-7124c2b0ed79"). InnerVolumeSpecName "kube-api-access-89m97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:36:37.329796 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.329713 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg5cb\" (UniqueName: \"kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb\") pod \"e2cb357f-6b83-4378-bb28-12800e77209a\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " Jul 31 20:36:37.329796 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.329782 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbqkf\" (UniqueName: \"kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf\") pod \"19cbeb13-aeb6-48a7-a272-4bc594475490\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " Jul 31 20:36:37.330014 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.329841 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca\") pod \"e2cb357f-6b83-4378-bb28-12800e77209a\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " Jul 31 20:36:37.330014 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.329900 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") pod \"e2cb357f-6b83-4378-bb28-12800e77209a\" (UID: \"e2cb357f-6b83-4378-bb28-12800e77209a\") " Jul 31 20:36:37.330014 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.329928 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume\") pod \"19cbeb13-aeb6-48a7-a272-4bc594475490\" (UID: \"19cbeb13-aeb6-48a7-a272-4bc594475490\") " Jul 31 20:36:37.330195 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.330177 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-plugins-config-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.330257 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.330192 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2cb357f-6b83-4378-bb28-12800e77209a" (UID: "e2cb357f-6b83-4378-bb28-12800e77209a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:36:37.330257 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.330200 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89m97\" (UniqueName: \"kubernetes.io/projected/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79-kube-api-access-89m97\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.330442 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.330415 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19cbeb13-aeb6-48a7-a272-4bc594475490" (UID: "19cbeb13-aeb6-48a7-a272-4bc594475490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:36:37.332008 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.331990 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2cb357f-6b83-4378-bb28-12800e77209a" (UID: "e2cb357f-6b83-4378-bb28-12800e77209a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:36:37.332296 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.332268 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf" (OuterVolumeSpecName: "kube-api-access-cbqkf") pod "19cbeb13-aeb6-48a7-a272-4bc594475490" (UID: "19cbeb13-aeb6-48a7-a272-4bc594475490"). InnerVolumeSpecName "kube-api-access-cbqkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:36:37.332419 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.332299 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb" (OuterVolumeSpecName: "kube-api-access-zg5cb") pod "e2cb357f-6b83-4378-bb28-12800e77209a" (UID: "e2cb357f-6b83-4378-bb28-12800e77209a"). InnerVolumeSpecName "kube-api-access-zg5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:36:37.430856 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.430819 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqkf\" (UniqueName: \"kubernetes.io/projected/19cbeb13-aeb6-48a7-a272-4bc594475490-kube-api-access-cbqkf\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.430856 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.430850 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb357f-6b83-4378-bb28-12800e77209a-openshift-service-ca\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.430856 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.430864 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2cb357f-6b83-4378-bb28-12800e77209a-maas-api-tls\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.431089 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.430876 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cbeb13-aeb6-48a7-a272-4bc594475490-plugins-config-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.431089 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.430888 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5cb\" (UniqueName: \"kubernetes.io/projected/e2cb357f-6b83-4378-bb28-12800e77209a-kube-api-access-zg5cb\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:36:37.692446 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.692357 2556 generic.go:356] "Generic (PLEG): container finished" podID="19cbeb13-aeb6-48a7-a272-4bc594475490" containerID="696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f" exitCode=0 Jul 31 20:36:37.692446 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.692398 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" Jul 31 20:36:37.692683 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.692451 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" event={"ID":"19cbeb13-aeb6-48a7-a272-4bc594475490","Type":"ContainerDied","Data":"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f"} Jul 31 20:36:37.692683 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.692504 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv" event={"ID":"19cbeb13-aeb6-48a7-a272-4bc594475490","Type":"ContainerDied","Data":"a68e47c0f3629855181b0a177a6c026527575091e5504cb54376588f277d460d"} Jul 31 20:36:37.692683 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.692533 2556 scope.go:117] "RemoveContainer" containerID="696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f" Jul 31 20:36:37.694029 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.694002 2556 generic.go:356] "Generic (PLEG): container finished" podID="e2cb357f-6b83-4378-bb28-12800e77209a" containerID="bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b" exitCode=0 Jul 31 20:36:37.694169 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.694072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" event={"ID":"e2cb357f-6b83-4378-bb28-12800e77209a","Type":"ContainerDied","Data":"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b"} Jul 31 20:36:37.694169 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.694123 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" event={"ID":"e2cb357f-6b83-4378-bb28-12800e77209a","Type":"ContainerDied","Data":"2f00cf4348b46e5146423e101cc5dbb03d7e4f858059cea793eebcd0ac24a4e7"} Jul 31 20:36:37.694169 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.694159 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx" Jul 31 20:36:37.695925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.695898 2556 generic.go:356] "Generic (PLEG): container finished" podID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" containerID="85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3" exitCode=0 Jul 31 20:36:37.696043 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.695930 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" event={"ID":"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79","Type":"ContainerDied","Data":"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3"} Jul 31 20:36:37.696043 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.695950 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" Jul 31 20:36:37.696043 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.695984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc" event={"ID":"6a48b1f9-9441-4f3b-96f6-7124c2b0ed79","Type":"ContainerDied","Data":"1a8f99fc464a1cf937ce7fde2daad3b0adfdf5428873c085c2cac406b26ef4a9"} Jul 31 20:36:37.704018 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.703725 2556 scope.go:117] "RemoveContainer" containerID="696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f" Jul 31 20:36:37.707770 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:36:37.707745 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f\": container with ID starting with 696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f not found: ID does not exist" containerID="696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f" Jul 31 20:36:37.707876 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.707782 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f"} err="failed to get container status \"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f\": rpc error: code = NotFound desc = could not find container \"696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f\": container with ID starting with 696115e8e17c8ddb1520163f40c7d8e2bf172b5925afa42f04bab825dcfa8b8f not found: ID does not exist" Jul 31 20:36:37.707876 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.707815 2556 scope.go:117] "RemoveContainer" containerID="bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b" Jul 31 20:36:37.718165 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.718141 2556 scope.go:117] "RemoveContainer" containerID="bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b" Jul 31 20:36:37.718478 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:36:37.718453 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b\": container with ID starting with bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b not found: ID does not exist" containerID="bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b" Jul 31 20:36:37.718585 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.718487 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b"} err="failed to get container status \"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b\": rpc error: code = NotFound desc = could not find container \"bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b\": container with ID starting with bb53874f88c831a5ff7770680a99c91bd2b25e7ba0b02731d846545dc22e212b not found: ID does not exist" Jul 31 20:36:37.718585 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.718506 2556 scope.go:117] "RemoveContainer" containerID="85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3" Jul 31 20:36:37.727259 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.727239 2556 scope.go:117] "RemoveContainer" containerID="85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3" Jul 31 20:36:37.727514 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:36:37.727493 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3\": container with ID starting with 85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3 not found: ID does not exist" containerID="85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3" Jul 31 20:36:37.727559 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.727525 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3"} err="failed to get container status \"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3\": rpc error: code = NotFound desc = could not find container \"85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3\": container with ID starting with 85fa68c6b1312e2e1045ba7547485fa048d85784187d7f24adfd8191d785fef3 not found: ID does not exist" Jul 31 20:36:37.747219 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.747192 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:36:37.755146 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.755114 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fa6acc8-84ffc4b489-gvrqx"] Jul 31 20:36:37.766152 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.766127 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:36:37.771898 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.771876 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fa6acc8-86bfbcb78b-55krc"] Jul 31 20:36:37.789925 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.789902 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:36:37.795606 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:37.795587 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fa6acc8-84fdcd9cbd-tlcvv"] Jul 31 20:36:39.167402 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:39.167369 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cbeb13-aeb6-48a7-a272-4bc594475490" path="/var/lib/kubelet/pods/19cbeb13-aeb6-48a7-a272-4bc594475490/volumes" Jul 31 20:36:39.167763 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:39.167720 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" path="/var/lib/kubelet/pods/6a48b1f9-9441-4f3b-96f6-7124c2b0ed79/volumes" Jul 31 20:36:39.168083 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:39.168068 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2cb357f-6b83-4378-bb28-12800e77209a" path="/var/lib/kubelet/pods/e2cb357f-6b83-4378-bb28-12800e77209a/volumes" Jul 31 20:36:41.110966 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.110931 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:36:41.111741 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111719 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cb357f-6b83-4378-bb28-12800e77209a" containerName="maas-api" Jul 31 20:36:41.111828 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111743 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cb357f-6b83-4378-bb28-12800e77209a" containerName="maas-api" Jul 31 20:36:41.111828 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111759 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" containerName="payload-processing" Jul 31 20:36:41.111828 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111768 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" containerName="payload-processing" Jul 31 20:36:41.111828 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111794 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cbeb13-aeb6-48a7-a272-4bc594475490" containerName="payload-pre-processing" Jul 31 20:36:41.111828 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111803 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cbeb13-aeb6-48a7-a272-4bc594475490" containerName="payload-pre-processing" Jul 31 20:36:41.112072 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111894 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cbeb13-aeb6-48a7-a272-4bc594475490" containerName="payload-pre-processing" Jul 31 20:36:41.112072 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111910 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cb357f-6b83-4378-bb28-12800e77209a" containerName="maas-api" Jul 31 20:36:41.112072 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.111919 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48b1f9-9441-4f3b-96f6-7124c2b0ed79" containerName="payload-processing" Jul 31 20:36:41.116277 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.116250 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.118543 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.118520 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8898d684-serving-cert\"" Jul 31 20:36:41.123660 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.123587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:36:41.263434 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.263398 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.263615 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.263466 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.263615 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.263497 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lsz\" (UniqueName: \"kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.364853 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.364769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.364853 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.364816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lsz\" (UniqueName: \"kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.364853 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.364854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.365450 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.365422 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.367161 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.367137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.372838 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.372816 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lsz\" (UniqueName: \"kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz\") pod \"maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.429010 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.428966 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:41.566736 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.566704 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:36:41.568008 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:36:41.567979 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d366098_239f_4a7c_8153_2336f5fd053b.slice/crio-7136db9f9d7ae2251e439fcd51f9a73137a7041b01f9fa61911e6cc52b2cdb05 WatchSource:0}: Error finding container 7136db9f9d7ae2251e439fcd51f9a73137a7041b01f9fa61911e6cc52b2cdb05: Status 404 returned error can't find the container with id 7136db9f9d7ae2251e439fcd51f9a73137a7041b01f9fa61911e6cc52b2cdb05 Jul 31 20:36:41.686130 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.686047 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:36:41.690730 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.690706 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.693606 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.693584 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8898d684\"" Jul 31 20:36:41.700737 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.700713 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:36:41.721307 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.721279 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" event={"ID":"5d366098-239f-4a7c-8153-2336f5fd053b","Type":"ContainerStarted","Data":"7136db9f9d7ae2251e439fcd51f9a73137a7041b01f9fa61911e6cc52b2cdb05"} Jul 31 20:36:41.727387 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.727364 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:36:41.730632 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.730617 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.732832 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.732816 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8898d684-dockercfg-gmsvm\"" Jul 31 20:36:41.740456 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.740436 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:36:41.766592 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.766567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnhg\" (UniqueName: \"kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.766712 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.766625 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.766712 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.766674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6z8g\" (UniqueName: \"kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.766805 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.766711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.867769 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.867733 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnhg\" (UniqueName: \"kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.867946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.867796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.867946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.867822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6z8g\" (UniqueName: \"kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.867946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.867841 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.868783 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.868761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.868844 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.868761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:41.876126 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.876099 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6z8g\" (UniqueName: \"kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g\") pod \"payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:41.876240 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:41.876106 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnhg\" (UniqueName: \"kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg\") pod \"payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:42.001912 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.001888 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:42.042242 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.042212 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:42.150644 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.150515 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:36:42.152204 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:36:42.152172 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f77d699_b8f6_459c_a673_746f3276a2cd.slice/crio-ff855a86f03610d8a95cace55742b3dd4ced0ece0d68ddcd3cc0638ae16dc2df WatchSource:0}: Error finding container ff855a86f03610d8a95cace55742b3dd4ced0ece0d68ddcd3cc0638ae16dc2df: Status 404 returned error can't find the container with id ff855a86f03610d8a95cace55742b3dd4ced0ece0d68ddcd3cc0638ae16dc2df Jul 31 20:36:42.206746 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.206721 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:36:42.207591 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:36:42.207569 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca57cc9e_19f8_401e_a3dd_bf07ff2a2f39.slice/crio-d0fcb685239e815744e685c2c07a61f8bf0247483c167055de8f4dc7c84fbe8c WatchSource:0}: Error finding container d0fcb685239e815744e685c2c07a61f8bf0247483c167055de8f4dc7c84fbe8c: Status 404 returned error can't find the container with id d0fcb685239e815744e685c2c07a61f8bf0247483c167055de8f4dc7c84fbe8c Jul 31 20:36:42.728788 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.728751 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" event={"ID":"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39","Type":"ContainerStarted","Data":"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c"} Jul 31 20:36:42.729004 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.728819 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:36:42.729004 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.728842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" event={"ID":"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39","Type":"ContainerStarted","Data":"d0fcb685239e815744e685c2c07a61f8bf0247483c167055de8f4dc7c84fbe8c"} Jul 31 20:36:42.730468 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.730430 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" event={"ID":"4f77d699-b8f6-459c-a673-746f3276a2cd","Type":"ContainerStarted","Data":"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3"} Jul 31 20:36:42.730601 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.730482 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:42.730601 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.730503 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" event={"ID":"4f77d699-b8f6-459c-a673-746f3276a2cd","Type":"ContainerStarted","Data":"ff855a86f03610d8a95cace55742b3dd4ced0ece0d68ddcd3cc0638ae16dc2df"} Jul 31 20:36:42.731967 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.731942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" event={"ID":"5d366098-239f-4a7c-8153-2336f5fd053b","Type":"ContainerStarted","Data":"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4"} Jul 31 20:36:42.732065 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.732051 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:42.744161 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.744121 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" podStartSLOduration=1.744108054 podStartE2EDuration="1.744108054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:36:42.743033924 +0000 UTC m=+2532.084617552" watchObservedRunningTime="2026-07-31 20:36:42.744108054 +0000 UTC m=+2532.085691726" Jul 31 20:36:42.759455 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.759409 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" podStartSLOduration=1.759393063 podStartE2EDuration="1.759393063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:36:42.757142079 +0000 UTC m=+2532.098725707" watchObservedRunningTime="2026-07-31 20:36:42.759393063 +0000 UTC m=+2532.100976688" Jul 31 20:36:42.775841 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:42.775803 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" podStartSLOduration=1.362099323 podStartE2EDuration="1.775786785s" totalImagesPullingTime="413.687462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:36:42.774907541 +0000 UTC m=+2532.116491168" watchObservedRunningTime="2026-07-31 20:36:42.775786785 +0000 UTC m=+2532.117370412" Jul 31 20:36:48.743102 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:48.743065 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:36:53.739999 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:53.739967 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:36:53.740393 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:36:53.740048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:39:31.355332 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:39:31.355215 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:39:31.361337 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:39:31.361300 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wncgg_806b7291-7bec-44fd-942d-8e0ec25d7ab8/ovn-acl-logging/0.log" Jul 31 20:40:41.033054 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.033017 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn"] Jul 31 20:40:41.036481 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.036463 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:41.038889 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.038863 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkrj7\"" Jul 31 20:40:41.050789 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.050766 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn"] Jul 31 20:40:41.157154 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.157114 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn\" (UID: \"fcf04c6d-8ace-4dad-a01e-dfe1286f451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:41.257971 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.257936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn\" (UID: \"fcf04c6d-8ace-4dad-a01e-dfe1286f451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:41.258609 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.258588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn\" (UID: \"fcf04c6d-8ace-4dad-a01e-dfe1286f451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:41.346027 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.345991 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:41.476474 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.476447 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn"] Jul 31 20:40:41.479510 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:40:41.479483 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf04c6d_8ace_4dad_a01e_dfe1286f451b.slice/crio-1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47 WatchSource:0}: Error finding container 1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47: Status 404 returned error can't find the container with id 1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47 Jul 31 20:40:41.944689 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.944599 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" event={"ID":"fcf04c6d-8ace-4dad-a01e-dfe1286f451b","Type":"ContainerStarted","Data":"a21771efcf34d88b0f29a1dc0b8145227edb0f99341c4be720afbeeabcb035c8"} Jul 31 20:40:41.944689 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.944656 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" event={"ID":"fcf04c6d-8ace-4dad-a01e-dfe1286f451b","Type":"ContainerStarted","Data":"1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47"} Jul 31 20:40:41.960467 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:41.960423 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" podStartSLOduration=0.960409427 podStartE2EDuration="960.409427ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:40:41.958065855 +0000 UTC m=+2771.299649481" watchObservedRunningTime="2026-07-31 20:40:41.960409427 +0000 UTC m=+2771.301993053" Jul 31 20:40:42.950873 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:42.950839 2556 generic.go:356] "Generic (PLEG): container finished" podID="fcf04c6d-8ace-4dad-a01e-dfe1286f451b" containerID="a21771efcf34d88b0f29a1dc0b8145227edb0f99341c4be720afbeeabcb035c8" exitCode=0 Jul 31 20:40:42.951260 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:42.950909 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" event={"ID":"fcf04c6d-8ace-4dad-a01e-dfe1286f451b","Type":"ContainerDied","Data":"a21771efcf34d88b0f29a1dc0b8145227edb0f99341c4be720afbeeabcb035c8"} Jul 31 20:40:44.087245 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.087223 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:44.281270 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.281182 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca\") pod \"fcf04c6d-8ace-4dad-a01e-dfe1286f451b\" (UID: \"fcf04c6d-8ace-4dad-a01e-dfe1286f451b\") " Jul 31 20:40:44.281558 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.281536 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcf04c6d-8ace-4dad-a01e-dfe1286f451b" (UID: "fcf04c6d-8ace-4dad-a01e-dfe1286f451b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:40:44.382629 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.382593 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf04c6d-8ace-4dad-a01e-dfe1286f451b-maas-api-service-ca\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:44.961497 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.961468 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" Jul 31 20:40:44.961677 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.961518 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn" event={"ID":"fcf04c6d-8ace-4dad-a01e-dfe1286f451b","Type":"ContainerDied","Data":"1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47"} Jul 31 20:40:44.961677 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:44.961598 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd2a35dc1d3c430369895169278090050d8bdd5a5b879754a93ee4659a8ce47" Jul 31 20:40:51.966738 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:51.966702 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:40:51.967524 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:51.967485 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" podUID="5d366098-239f-4a7c-8153-2336f5fd053b" containerName="maas-api" containerID="cri-o://04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4" gracePeriod=30 Jul 31 20:40:52.021676 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.021646 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:40:52.021889 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.021863 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" podUID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" containerName="payload-processing" containerID="cri-o://bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c" gracePeriod=30 Jul 31 20:40:52.030474 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.030425 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:40:52.030708 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.030675 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" podUID="4f77d699-b8f6-459c-a673-746f3276a2cd" containerName="payload-pre-processing" containerID="cri-o://5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3" gracePeriod=30 Jul 31 20:40:52.168296 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.168229 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn"] Jul 31 20:40:52.172448 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.172351 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8898d684-7f7fn"] Jul 31 20:40:52.324397 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.324372 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:40:52.327343 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.327307 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:40:52.331147 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.331129 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:40:52.346144 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346108 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4lsz\" (UniqueName: \"kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz\") pod \"5d366098-239f-4a7c-8153-2336f5fd053b\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " Jul 31 20:40:52.346263 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346151 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls\") pod \"5d366098-239f-4a7c-8153-2336f5fd053b\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " Jul 31 20:40:52.346263 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346172 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6z8g\" (UniqueName: \"kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g\") pod \"4f77d699-b8f6-459c-a673-746f3276a2cd\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " Jul 31 20:40:52.346263 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346205 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca\") pod \"5d366098-239f-4a7c-8153-2336f5fd053b\" (UID: \"5d366098-239f-4a7c-8153-2336f5fd053b\") " Jul 31 20:40:52.346263 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346237 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume\") pod \"4f77d699-b8f6-459c-a673-746f3276a2cd\" (UID: \"4f77d699-b8f6-459c-a673-746f3276a2cd\") " Jul 31 20:40:52.346511 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346278 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume\") pod \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " Jul 31 20:40:52.346511 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346354 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxnhg\" (UniqueName: \"kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg\") pod \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\" (UID: \"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39\") " Jul 31 20:40:52.346628 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346579 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d366098-239f-4a7c-8153-2336f5fd053b" (UID: "5d366098-239f-4a7c-8153-2336f5fd053b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:40:52.346786 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.346754 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" (UID: "ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:40:52.347179 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.347155 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f77d699-b8f6-459c-a673-746f3276a2cd" (UID: "4f77d699-b8f6-459c-a673-746f3276a2cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:40:52.348504 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.348449 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g" (OuterVolumeSpecName: "kube-api-access-g6z8g") pod "4f77d699-b8f6-459c-a673-746f3276a2cd" (UID: "4f77d699-b8f6-459c-a673-746f3276a2cd"). InnerVolumeSpecName "kube-api-access-g6z8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:40:52.348910 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.348885 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg" (OuterVolumeSpecName: "kube-api-access-bxnhg") pod "ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" (UID: "ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39"). InnerVolumeSpecName "kube-api-access-bxnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:40:52.348984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.348906 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz" (OuterVolumeSpecName: "kube-api-access-d4lsz") pod "5d366098-239f-4a7c-8153-2336f5fd053b" (UID: "5d366098-239f-4a7c-8153-2336f5fd053b"). InnerVolumeSpecName "kube-api-access-d4lsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:40:52.348984 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.348952 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d366098-239f-4a7c-8153-2336f5fd053b" (UID: "5d366098-239f-4a7c-8153-2336f5fd053b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:40:52.447495 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447463 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnhg\" (UniqueName: \"kubernetes.io/projected/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-kube-api-access-bxnhg\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447495 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447490 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4lsz\" (UniqueName: \"kubernetes.io/projected/5d366098-239f-4a7c-8153-2336f5fd053b-kube-api-access-d4lsz\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447504 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d366098-239f-4a7c-8153-2336f5fd053b-maas-api-tls\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447515 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6z8g\" (UniqueName: \"kubernetes.io/projected/4f77d699-b8f6-459c-a673-746f3276a2cd-kube-api-access-g6z8g\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447524 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d366098-239f-4a7c-8153-2336f5fd053b-openshift-service-ca\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447532 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f77d699-b8f6-459c-a673-746f3276a2cd-plugins-config-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:52.447672 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:52.447539 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39-plugins-config-volume\") on node \"ip-10-0-143-178.ec2.internal\" DevicePath \"\"" Jul 31 20:40:53.004088 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.004050 2556 generic.go:356] "Generic (PLEG): container finished" podID="5d366098-239f-4a7c-8153-2336f5fd053b" containerID="04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4" exitCode=0 Jul 31 20:40:53.004477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.004096 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" Jul 31 20:40:53.004477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.004132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" event={"ID":"5d366098-239f-4a7c-8153-2336f5fd053b","Type":"ContainerDied","Data":"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4"} Jul 31 20:40:53.004477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.004186 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8" event={"ID":"5d366098-239f-4a7c-8153-2336f5fd053b","Type":"ContainerDied","Data":"7136db9f9d7ae2251e439fcd51f9a73137a7041b01f9fa61911e6cc52b2cdb05"} Jul 31 20:40:53.004477 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.004208 2556 scope.go:117] "RemoveContainer" containerID="04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4" Jul 31 20:40:53.005727 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.005705 2556 generic.go:356] "Generic (PLEG): container finished" podID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" containerID="bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c" exitCode=0 Jul 31 20:40:53.005813 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.005758 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" Jul 31 20:40:53.005813 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.005756 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" event={"ID":"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39","Type":"ContainerDied","Data":"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c"} Jul 31 20:40:53.005813 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.005809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl" event={"ID":"ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39","Type":"ContainerDied","Data":"d0fcb685239e815744e685c2c07a61f8bf0247483c167055de8f4dc7c84fbe8c"} Jul 31 20:40:53.007272 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.007240 2556 generic.go:356] "Generic (PLEG): container finished" podID="4f77d699-b8f6-459c-a673-746f3276a2cd" containerID="5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3" exitCode=0 Jul 31 20:40:53.007389 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.007286 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" Jul 31 20:40:53.007389 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.007287 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" event={"ID":"4f77d699-b8f6-459c-a673-746f3276a2cd","Type":"ContainerDied","Data":"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3"} Jul 31 20:40:53.007389 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.007381 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4" event={"ID":"4f77d699-b8f6-459c-a673-746f3276a2cd","Type":"ContainerDied","Data":"ff855a86f03610d8a95cace55742b3dd4ced0ece0d68ddcd3cc0638ae16dc2df"} Jul 31 20:40:53.015456 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.015437 2556 scope.go:117] "RemoveContainer" containerID="04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4" Jul 31 20:40:53.015777 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:40:53.015753 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4\": container with ID starting with 04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4 not found: ID does not exist" containerID="04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4" Jul 31 20:40:53.015871 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.015789 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4"} err="failed to get container status \"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4\": rpc error: code = NotFound desc = could not find container \"04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4\": container with ID starting with 04bd4346287d831292a42ae55094b170e03c27690f1b2c724caae56f30cd9af4 not found: ID does not exist" Jul 31 20:40:53.015871 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.015812 2556 scope.go:117] "RemoveContainer" containerID="bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c" Jul 31 20:40:53.023791 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.023772 2556 scope.go:117] "RemoveContainer" containerID="bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c" Jul 31 20:40:53.024053 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:40:53.024035 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c\": container with ID starting with bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c not found: ID does not exist" containerID="bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c" Jul 31 20:40:53.024141 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.024057 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c"} err="failed to get container status \"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c\": rpc error: code = NotFound desc = could not find container \"bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c\": container with ID starting with bdc098a761e1d7e7e6f7e4a5c3f67ea054ba908ebecab9bda26602184f539a0c not found: ID does not exist" Jul 31 20:40:53.024141 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.024072 2556 scope.go:117] "RemoveContainer" containerID="5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3" Jul 31 20:40:53.031126 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.031102 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:40:53.033401 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.033385 2556 scope.go:117] "RemoveContainer" containerID="5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3" Jul 31 20:40:53.033758 ip-10-0-143-178 kubenswrapper[2556]: E0731 20:40:53.033715 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3\": container with ID starting with 5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3 not found: ID does not exist" containerID="5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3" Jul 31 20:40:53.033838 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.033749 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3"} err="failed to get container status \"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3\": rpc error: code = NotFound desc = could not find container \"5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3\": container with ID starting with 5e9aaeaf02f435fb78a31b666b2baf9063f011160263ae1be016d9f5546143b3 not found: ID does not exist" Jul 31 20:40:53.039241 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.039219 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8898d684-7db47d8d46-tdvw8"] Jul 31 20:40:53.049809 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.049787 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:40:53.052383 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.052359 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8898d684-6947b9559c-kmxfl"] Jul 31 20:40:53.062035 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.062013 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:40:53.068091 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.068072 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8898d684-bbc9f56-xnrr4"] Jul 31 20:40:53.166100 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.166072 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f77d699-b8f6-459c-a673-746f3276a2cd" path="/var/lib/kubelet/pods/4f77d699-b8f6-459c-a673-746f3276a2cd/volumes" Jul 31 20:40:53.166481 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.166467 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d366098-239f-4a7c-8153-2336f5fd053b" path="/var/lib/kubelet/pods/5d366098-239f-4a7c-8153-2336f5fd053b/volumes" Jul 31 20:40:53.166773 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.166762 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" path="/var/lib/kubelet/pods/ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39/volumes" Jul 31 20:40:53.167027 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:53.167016 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf04c6d-8ace-4dad-a01e-dfe1286f451b" path="/var/lib/kubelet/pods/fcf04c6d-8ace-4dad-a01e-dfe1286f451b/volumes" Jul 31 20:40:58.743043 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:58.742965 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zj2jd_1922c780-f7c8-4e28-81ed-27375ff15224/manager/0.log" Jul 31 20:40:58.870769 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:58.870737 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-79mbv_b74953c6-c6d3-47ff-83a5-8fbdac3d2c96/manager/0.log" Jul 31 20:40:58.994417 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:58.994345 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-777c587c75-l75cv_37769134-9c91-44af-b193-c5b1d046b730/manager/0.log" Jul 31 20:40:59.118567 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:59.118541 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8kh82_70fc3499-3444-49f7-a1c9-f0b83e098aae/server/0.log" Jul 31 20:40:59.492681 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:59.492648 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675f995665-8s4gc_c26f3a50-49d3-47df-bd54-bfb9d76361fe/copy-manifests/0.log" Jul 31 20:40:59.498789 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:40:59.498765 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675f995665-8s4gc_c26f3a50-49d3-47df-bd54-bfb9d76361fe/manager/0.log" Jul 31 20:41:00.006749 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:00.006721 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c4944758f-r5l5m_a9883ae5-2328-48fe-aa30-c3550ea96097/maas-api/0.log" Jul 31 20:41:00.128007 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:00.127983 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758815-whvk2_c6b4d9b3-0f25-433b-a87e-3108852a1170/cleanup/0.log" Jul 31 20:41:00.372684 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:00.372658 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tzbhg_59017ab3-9972-49fd-9378-dcdb355784e9/postgres/0.log" Jul 31 20:41:01.334391 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:01.334359 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-qnwrs_592a1350-8bf6-490b-8509-75502d3b1d31/manager/0.log" Jul 31 20:41:01.457005 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:01.456977 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-d8xgh_dd0c848b-6709-43cb-90b2-787ae5985fab/manager/0.log" Jul 31 20:41:01.701114 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:01.701037 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-hn97d_2fe4b7c5-38f7-4154-870a-aa040c467a6e/manager/0.log" Jul 31 20:41:01.943689 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:01.943658 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-gjzfw_f8a156ed-c678-4dde-86b6-7e81aa6fe149/manager/0.log" Jul 31 20:41:02.572301 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:02.572269 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f45495bd6-4644m_91601de3-a95e-4f35-81b6-a51c2b6d0b1a/kube-auth-proxy/0.log" Jul 31 20:41:02.941578 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:02.941502 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-b47wv_eedaa52f-8b8e-462f-83e7-fcf362f38826/payload-pre-processing/0.log" Jul 31 20:41:03.063257 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:03.063231 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2p7zz_a2e3916c-31ab-4d2e-9bd7-19ec0eab3033/payload-processing/0.log" Jul 31 20:41:03.547178 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:03.547145 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jg5p6_d8bee38b-dddd-43cf-b959-5409f024f3b3/main/0.log" Jul 31 20:41:03.789800 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:03.789770 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-57cwg_156b27d1-1e97-485e-834a-1a38526461a6/main/0.log" Jul 31 20:41:04.039619 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:04.039576 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hnxwj_846dfaf0-8e28-4ef5-803a-9b61d86b013a/main/0.log" Jul 31 20:41:10.261429 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261393 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pkxh/must-gather-kjnzn"] Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261721 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f77d699-b8f6-459c-a673-746f3276a2cd" containerName="payload-pre-processing" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261733 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f77d699-b8f6-459c-a673-746f3276a2cd" containerName="payload-pre-processing" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261749 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" containerName="payload-processing" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261769 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" containerName="payload-processing" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261778 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf04c6d-8ace-4dad-a01e-dfe1286f451b" containerName="revoke-keys" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261783 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf04c6d-8ace-4dad-a01e-dfe1286f451b" containerName="revoke-keys" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261797 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d366098-239f-4a7c-8153-2336f5fd053b" containerName="maas-api" Jul 31 20:41:10.261823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261801 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d366098-239f-4a7c-8153-2336f5fd053b" containerName="maas-api" Jul 31 20:41:10.262061 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261850 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f77d699-b8f6-459c-a673-746f3276a2cd" containerName="payload-pre-processing" Jul 31 20:41:10.262061 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261859 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d366098-239f-4a7c-8153-2336f5fd053b" containerName="maas-api" Jul 31 20:41:10.262061 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261865 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca57cc9e-19f8-401e-a3dd-bf07ff2a2f39" containerName="payload-processing" Jul 31 20:41:10.262061 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.261872 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf04c6d-8ace-4dad-a01e-dfe1286f451b" containerName="revoke-keys" Jul 31 20:41:10.264946 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.264924 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.267501 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.267478 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pkxh\"/\"kube-root-ca.crt\"" Jul 31 20:41:10.268137 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.268120 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pkxh\"/\"openshift-service-ca.crt\"" Jul 31 20:41:10.268217 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.268122 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5pkxh\"/\"default-dockercfg-8b7kl\"" Jul 31 20:41:10.272010 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.271991 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pkxh/must-gather-kjnzn"] Jul 31 20:41:10.389831 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.389798 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-must-gather-output\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.389999 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.389857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8b9s\" (UniqueName: \"kubernetes.io/projected/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-kube-api-access-q8b9s\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.490767 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.490736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8b9s\" (UniqueName: \"kubernetes.io/projected/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-kube-api-access-q8b9s\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.490928 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.490785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-must-gather-output\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.491093 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.491075 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-must-gather-output\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.501615 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.501592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8b9s\" (UniqueName: \"kubernetes.io/projected/046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c-kube-api-access-q8b9s\") pod \"must-gather-kjnzn\" (UID: \"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c\") " pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.574891 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.574854 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" Jul 31 20:41:10.703748 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.703719 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pkxh/must-gather-kjnzn"] Jul 31 20:41:10.707051 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:41:10.707020 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod046e8f0d_3e11_4ef3_ba7b_8aa7ccc7482c.slice/crio-0be6e089ec89e4bd7c17f15c470d79fe7ebab90ff0b296e680996fe5ad4d81e6 WatchSource:0}: Error finding container 0be6e089ec89e4bd7c17f15c470d79fe7ebab90ff0b296e680996fe5ad4d81e6: Status 404 returned error can't find the container with id 0be6e089ec89e4bd7c17f15c470d79fe7ebab90ff0b296e680996fe5ad4d81e6 Jul 31 20:41:10.709051 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:10.709036 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:41:11.108026 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:11.107993 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" event={"ID":"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c","Type":"ContainerStarted","Data":"0be6e089ec89e4bd7c17f15c470d79fe7ebab90ff0b296e680996fe5ad4d81e6"} Jul 31 20:41:12.122222 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:12.122179 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" event={"ID":"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c","Type":"ContainerStarted","Data":"d2d826be87801d5576e98e0a36aaa62a7d0369b0446f0bb4435ecd263584dd28"} Jul 31 20:41:12.122804 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:12.122237 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" event={"ID":"046e8f0d-3e11-4ef3-ba7b-8aa7ccc7482c","Type":"ContainerStarted","Data":"e9c786141060237c8612b6304f73ec74b0cef990b790c51d9dba7aef642496ab"} Jul 31 20:41:12.138271 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:12.138211 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5pkxh/must-gather-kjnzn" podStartSLOduration=1.321388502 podStartE2EDuration="2.138196621s" totalImagesPullingTime="816.808119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:41:12.1373054 +0000 UTC m=+2801.478889037" watchObservedRunningTime="2026-07-31 20:41:12.138196621 +0000 UTC m=+2801.479780247" Jul 31 20:41:13.050386 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:13.050354 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hf82d_e9584fe9-248f-4aa7-8dd0-2633a9bf857f/global-pull-secret-syncer/0.log" Jul 31 20:41:13.137871 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:13.137837 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c9t5t_4b788b4e-4803-4515-bba2-39b870e004f7/konnectivity-agent/0.log" Jul 31 20:41:13.209940 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:13.209909 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-178.ec2.internal_2de2a3e1acdc244afa40a51cd31b1009/haproxy/0.log" Jul 31 20:41:18.486814 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:18.486754 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-qnwrs_592a1350-8bf6-490b-8509-75502d3b1d31/manager/0.log" Jul 31 20:41:18.511871 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:18.511835 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-d8xgh_dd0c848b-6709-43cb-90b2-787ae5985fab/manager/0.log" Jul 31 20:41:18.660552 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:18.660515 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-hn97d_2fe4b7c5-38f7-4154-870a-aa040c467a6e/manager/0.log" Jul 31 20:41:18.750975 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:18.750707 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-gjzfw_f8a156ed-c678-4dde-86b6-7e81aa6fe149/manager/0.log" Jul 31 20:41:20.532960 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:20.532871 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hrxj_ceabba42-5a1a-4eb3-8d94-6cea37e46571/node-exporter/0.log" Jul 31 20:41:20.559685 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:20.559645 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hrxj_ceabba42-5a1a-4eb3-8d94-6cea37e46571/kube-rbac-proxy/0.log" Jul 31 20:41:20.583212 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:20.583178 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5hrxj_ceabba42-5a1a-4eb3-8d94-6cea37e46571/init-textfile/0.log" Jul 31 20:41:22.647132 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.647096 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2"] Jul 31 20:41:22.652218 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.652187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.661387 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.661355 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2"] Jul 31 20:41:22.815151 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.815115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-sys\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.815442 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.815420 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-lib-modules\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.815658 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.815637 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-proc\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.815788 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.815772 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gk92\" (UniqueName: \"kubernetes.io/projected/26dcbd28-2df2-45c4-8424-a4c17835e66c-kube-api-access-7gk92\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.815916 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.815901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-podres\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.916988 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.916916 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-proc\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.916988 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.916952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gk92\" (UniqueName: \"kubernetes.io/projected/26dcbd28-2df2-45c4-8424-a4c17835e66c-kube-api-access-7gk92\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.916988 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.916971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-podres\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917008 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-sys\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917031 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-lib-modules\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-proc\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-sys\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-lib-modules\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.917258 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.917167 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dcbd28-2df2-45c4-8424-a4c17835e66c-podres\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.924581 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.924549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gk92\" (UniqueName: \"kubernetes.io/projected/26dcbd28-2df2-45c4-8424-a4c17835e66c-kube-api-access-7gk92\") pod \"perf-node-gather-daemonset-pzrg2\" (UID: \"26dcbd28-2df2-45c4-8424-a4c17835e66c\") " pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:22.971265 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:22.971229 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:23.165849 ip-10-0-143-178 kubenswrapper[2556]: W0731 20:41:23.165816 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26dcbd28_2df2_45c4_8424_a4c17835e66c.slice/crio-b20fa9c3faa0d6e92f984f721f04b7ebf6c96fd8d525b18887b7d1cced21a134 WatchSource:0}: Error finding container b20fa9c3faa0d6e92f984f721f04b7ebf6c96fd8d525b18887b7d1cced21a134: Status 404 returned error can't find the container with id b20fa9c3faa0d6e92f984f721f04b7ebf6c96fd8d525b18887b7d1cced21a134 Jul 31 20:41:23.169142 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:23.169027 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2"] Jul 31 20:41:23.208450 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:23.208415 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" event={"ID":"26dcbd28-2df2-45c4-8424-a4c17835e66c","Type":"ContainerStarted","Data":"b20fa9c3faa0d6e92f984f721f04b7ebf6c96fd8d525b18887b7d1cced21a134"} Jul 31 20:41:24.218057 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:24.218017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" event={"ID":"26dcbd28-2df2-45c4-8424-a4c17835e66c","Type":"ContainerStarted","Data":"7e795bff0a0e3fdc5f5acb8c743ae9ecfd25291e4c1a5262679494691521fd3f"} Jul 31 20:41:24.218492 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:24.218089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:24.234302 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:24.234245 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" podStartSLOduration=2.234225495 podStartE2EDuration="2.234225495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:41:24.232685917 +0000 UTC m=+2813.574269544" watchObservedRunningTime="2026-07-31 20:41:24.234225495 +0000 UTC m=+2813.575809123" Jul 31 20:41:25.087155 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:25.087112 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48wns_57414e35-918a-41db-a3ad-cd04428a36dd/dns/0.log" Jul 31 20:41:25.108450 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:25.108425 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48wns_57414e35-918a-41db-a3ad-cd04428a36dd/kube-rbac-proxy/0.log" Jul 31 20:41:25.239463 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:25.239433 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c2l2b_09eabf70-c174-4ddd-958c-8b43daed39dd/dns-node-resolver/0.log" Jul 31 20:41:25.768363 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:25.768332 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5d27_0af497c1-b530-46e6-861d-8324d9d3d829/node-ca/0.log" Jul 31 20:41:26.836971 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:26.836943 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f45495bd6-4644m_91601de3-a95e-4f35-81b6-a51c2b6d0b1a/kube-auth-proxy/0.log" Jul 31 20:41:26.931947 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:26.931907 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-b47wv_eedaa52f-8b8e-462f-83e7-fcf362f38826/payload-pre-processing/0.log" Jul 31 20:41:26.966166 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:26.966104 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2p7zz_a2e3916c-31ab-4d2e-9bd7-19ec0eab3033/payload-processing/0.log" Jul 31 20:41:27.623669 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:27.623641 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgjn8_04080c1f-cb16-4c9c-96f3-baf406112fc2/serve-healthcheck-canary/0.log" Jul 31 20:41:28.196075 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:28.196046 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqrm2_81566aae-bac4-44f8-b062-5e88eeaa90a3/kube-rbac-proxy/0.log" Jul 31 20:41:28.215112 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:28.215080 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqrm2_81566aae-bac4-44f8-b062-5e88eeaa90a3/exporter/0.log" Jul 31 20:41:28.235821 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:28.235798 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqrm2_81566aae-bac4-44f8-b062-5e88eeaa90a3/extractor/0.log" Jul 31 20:41:30.232844 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.232818 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5pkxh/perf-node-gather-daemonset-pzrg2" Jul 31 20:41:30.382414 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.382380 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zj2jd_1922c780-f7c8-4e28-81ed-27375ff15224/manager/0.log" Jul 31 20:41:30.494305 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.494205 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-79mbv_b74953c6-c6d3-47ff-83a5-8fbdac3d2c96/manager/0.log" Jul 31 20:41:30.634638 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.634577 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-777c587c75-l75cv_37769134-9c91-44af-b193-c5b1d046b730/manager/0.log" Jul 31 20:41:30.655060 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.655036 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8kh82_70fc3499-3444-49f7-a1c9-f0b83e098aae/server/0.log" Jul 31 20:41:30.775133 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.775069 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675f995665-8s4gc_c26f3a50-49d3-47df-bd54-bfb9d76361fe/manager/0.log" Jul 31 20:41:30.796379 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:30.796355 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675f995665-8s4gc_c26f3a50-49d3-47df-bd54-bfb9d76361fe/copy-manifests/0.log" Jul 31 20:41:32.971543 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:32.971481 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f45495bd6-4644m_91601de3-a95e-4f35-81b6-a51c2b6d0b1a/kube-auth-proxy/0.log" Jul 31 20:41:33.056419 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:33.056389 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-b47wv_eedaa52f-8b8e-462f-83e7-fcf362f38826/payload-pre-processing/0.log" Jul 31 20:41:33.078769 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:33.078742 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2p7zz_a2e3916c-31ab-4d2e-9bd7-19ec0eab3033/payload-processing/0.log" Jul 31 20:41:34.331192 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:34.331145 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f45495bd6-4644m_91601de3-a95e-4f35-81b6-a51c2b6d0b1a/kube-auth-proxy/0.log" Jul 31 20:41:34.409020 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:34.408989 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-b47wv_eedaa52f-8b8e-462f-83e7-fcf362f38826/payload-pre-processing/0.log" Jul 31 20:41:34.433514 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:34.433478 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2p7zz_a2e3916c-31ab-4d2e-9bd7-19ec0eab3033/payload-processing/0.log" Jul 31 20:41:38.770823 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.770795 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/kube-multus-additional-cni-plugins/0.log" Jul 31 20:41:38.792541 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.792512 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/egress-router-binary-copy/0.log" Jul 31 20:41:38.812693 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.812671 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/cni-plugins/0.log" Jul 31 20:41:38.832506 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.832483 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/bond-cni-plugin/0.log" Jul 31 20:41:38.851213 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.851194 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/routeoverride-cni/0.log" Jul 31 20:41:38.869863 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.869844 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/whereabouts-cni-bincopy/0.log" Jul 31 20:41:38.889406 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:38.889386 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6h2z_e4e72d7e-54cd-404c-817c-eb5baa46fe50/whereabouts-cni/0.log" Jul 31 20:41:39.073705 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:39.073680 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fn5g2_52ae78ac-54e5-49b8-8447-8f7f17fd318a/kube-multus/0.log" Jul 31 20:41:39.190452 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:39.190425 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qn4bd_7a481939-8e1f-4acd-9361-e7095e8d52f3/network-metrics-daemon/0.log" Jul 31 20:41:39.210560 ip-10-0-143-178 kubenswrapper[2556]: I0731 20:41:39.210541 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qn4bd_7a481939-8e1f-4acd-9361-e7095e8d52f3/kube-rbac-proxy/0.log"