Jul 06 20:24:17.337471 ip-10-0-134-98 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:24:18.258805 ip-10-0-134-98 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:24:18.258805 ip-10-0-134-98 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:24:18.258805 ip-10-0-134-98 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:24:18.258805 ip-10-0-134-98 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:24:18.258805 ip-10-0-134-98 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:24:18.261760 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.261592 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268182 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268202 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268208 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268212 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268216 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.268207 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268238 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268243 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268248 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268252 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268256 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268268 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268274 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268278 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268282 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268287 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268291 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268295 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268300 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268304 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268308 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268312 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268316 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268320 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268325 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.268566 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268329 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268334 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268338 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268342 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268347 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268352 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268356 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268361 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268366 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268372 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268376 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268381 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268385 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268390 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268395 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268400 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268405 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268410 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268415 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268420 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.269406 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268425 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268430 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268435 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268439 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268443 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268448 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268452 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268457 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268463 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268471 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268477 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268482 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268486 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268491 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268496 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268502 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268507 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268512 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268517 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268522 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.270320 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268527 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268531 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268536 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268541 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268547 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268554 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268561 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268567 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268574 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268580 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268586 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268591 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268596 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268604 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268608 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268613 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268617 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268623 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268628 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.271169 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268632 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268637 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268642 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268646 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268653 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268658 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268663 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268668 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268672 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268678 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268683 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268688 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268693 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268698 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268703 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.268708 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.269994 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270005 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270010 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270015 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.272046 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270021 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270026 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270032 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270039 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270048 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270053 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270059 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270064 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270069 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270075 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270080 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270085 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270089 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270094 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270098 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270105 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270112 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270117 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270121 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.272962 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270126 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270131 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270136 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270140 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270145 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270149 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270155 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270159 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270163 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270168 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270173 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270178 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270185 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270191 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270196 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270200 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270205 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270209 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270214 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270219 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.273797 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270246 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270251 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270255 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270261 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270266 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270270 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270275 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270279 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270285 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270290 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270294 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270299 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270303 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270308 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270313 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270317 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270321 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270325 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270329 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270335 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.274422 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270339 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270344 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270350 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270354 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270359 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270363 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270368 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270372 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270377 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270383 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270388 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270392 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270398 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270402 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270408 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270413 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270418 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270422 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270427 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270432 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.275195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270436 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270440 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270445 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270449 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270454 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270458 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270463 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270467 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270472 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270476 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270481 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270486 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270491 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270496 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270501 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.270506 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271196 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271204 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271209 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271214 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.275756 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271219 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271247 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271251 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271256 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271262 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271266 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271271 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271276 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271281 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271286 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271290 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271295 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271299 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271304 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271308 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271314 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271321 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271325 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271330 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.276418 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271334 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271339 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271346 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271352 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271357 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271366 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271372 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271377 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271382 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271387 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271392 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271397 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271401 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271405 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271410 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271417 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271421 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271426 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271431 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271435 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.277026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271440 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271444 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271448 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271453 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271457 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271462 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271467 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271471 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271476 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271480 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271484 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271489 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271493 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271498 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271502 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271507 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271512 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271516 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271522 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.277595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271526 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271531 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271535 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271540 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271544 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271548 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271553 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271557 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271561 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271566 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271570 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271574 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271578 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271583 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271592 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271597 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271602 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271607 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271612 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.278093 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271616 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271620 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271624 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271628 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271632 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271637 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271641 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271646 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271651 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271656 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271660 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271664 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271670 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271676 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271681 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271685 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271690 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.271695 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271816 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271828 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271838 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:24:18.278639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271845 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271853 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271858 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271865 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271873 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271879 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271884 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271891 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271897 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271902 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271908 2562 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271913 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271918 2562 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271924 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271929 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271937 2562 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271942 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271949 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271955 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271961 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271968 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271974 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271979 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:24:18.279183 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271985 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271991 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.271997 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272002 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272007 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272013 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272021 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272026 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272031 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272036 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272042 2562 flags.go:64] FLAG: --enable-server="true" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272047 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272054 2562 flags.go:64] FLAG: --event-burst="100" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272059 2562 flags.go:64] FLAG: --event-qps="50" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272064 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272069 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272074 2562 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272082 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272087 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272092 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272097 2562 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272103 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272108 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272113 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272118 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:24:18.279892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272123 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272129 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272134 2562 flags.go:64] FLAG: --feature-gates="" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272141 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272146 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272152 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272158 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272163 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272168 2562 flags.go:64] FLAG: --help="false" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272174 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272180 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272186 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272191 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272197 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272203 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272208 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272214 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272236 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272242 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272247 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272253 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272258 2562 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272263 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272268 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:24:18.280805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272273 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272290 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272296 2562 flags.go:64] FLAG: --lock-file="" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272301 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272307 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272312 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272347 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272355 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272361 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272366 2562 flags.go:64] FLAG: --logging-format="text" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272372 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272378 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272383 2562 flags.go:64] FLAG: --manifest-url="" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272389 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272397 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272405 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272412 2562 flags.go:64] FLAG: --max-pods="110" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272418 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272423 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272430 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272435 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272440 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272445 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272451 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272464 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:24:18.281811 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272469 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272475 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272480 2562 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272485 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272494 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272499 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272505 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272510 2562 flags.go:64] FLAG: --port="10250" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272517 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272523 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0015c66d053c82fc9" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272529 2562 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272534 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272540 2562 flags.go:64] FLAG: --register-node="true" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272544 2562 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272551 2562 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272556 2562 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272561 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272566 2562 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272572 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272578 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272583 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272589 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272594 2562 flags.go:64] FLAG: --runonce="false" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272601 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272607 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:24:18.282958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272612 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272617 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272623 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272629 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272635 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272640 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272645 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272650 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272655 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272660 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272666 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272671 2562 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272676 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272687 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272692 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272699 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272707 2562 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272712 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272718 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272723 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272728 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272733 2562 flags.go:64] FLAG: --v="2" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272740 2562 flags.go:64] FLAG: --version="false" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272747 2562 flags.go:64] FLAG: --vmodule="" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272753 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:24:18.283828 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.272760 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272969 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272976 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272981 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272986 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272993 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.272998 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273003 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273009 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273014 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273020 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273025 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273029 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273034 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273038 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273044 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273048 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273053 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273058 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273062 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.284958 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273067 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273072 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273079 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273084 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273089 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273094 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273098 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273105 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273112 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273117 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273121 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273128 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273134 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273139 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273145 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273151 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273156 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273162 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273167 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.285803 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273171 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273176 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273181 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273186 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273191 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273196 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273200 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273205 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273210 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273214 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273218 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273240 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273245 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273249 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273256 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273264 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273269 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273274 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273279 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.286383 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273284 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273288 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273293 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273298 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273303 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273309 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273313 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273318 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273323 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273327 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273332 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273339 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273343 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273348 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273352 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273357 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273362 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273366 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273372 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273377 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.287204 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273381 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273386 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273391 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273395 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273400 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273405 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273409 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273414 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273421 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273425 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273430 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273434 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273439 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273444 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273448 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273453 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273457 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273461 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273466 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273470 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.288135 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273474 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.273479 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.273488 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.283068 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.283093 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.283901 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.283916 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284019 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284028 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284034 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284039 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284044 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284049 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284055 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.288750 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284062 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284067 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284071 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284077 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284082 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284086 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284091 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284096 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284101 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284106 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284110 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284115 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284119 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284127 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284134 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284140 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284145 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284150 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284155 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284160 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.289215 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284166 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284171 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284176 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284180 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284185 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284190 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284194 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284199 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284203 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284208 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284213 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284218 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284239 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284244 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284248 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284253 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284257 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284262 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284267 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284273 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.289790 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284277 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284282 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284287 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284291 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284297 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284304 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284310 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284316 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284321 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284325 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284331 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284335 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284342 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284348 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284354 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284359 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284364 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284369 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284374 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.290398 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284378 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284383 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284387 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284392 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284397 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284401 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284406 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284411 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284415 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284420 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284424 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284429 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284434 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284440 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284445 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284450 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284455 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284459 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284464 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284469 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.290903 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284473 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284478 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284482 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284487 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284492 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284497 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284502 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284507 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284511 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284516 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284521 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284525 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284530 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.284539 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284777 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:24:18.291569 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284788 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284794 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284799 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284804 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284808 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284813 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284817 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284822 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284827 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284831 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284836 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284841 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284846 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284853 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284860 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284866 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284870 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284875 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284880 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284885 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:24:18.291964 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284889 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284895 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284899 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284904 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284908 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284913 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284917 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284922 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284927 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284931 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284936 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284940 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284945 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284949 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284954 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284959 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284963 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284967 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284971 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284975 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:24:18.292521 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284980 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284984 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284988 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284993 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.284998 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285003 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285008 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285013 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285017 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285022 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285027 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285032 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285039 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285045 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285050 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285054 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285059 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285064 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285068 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:24:18.293063 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285073 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285078 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285083 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285087 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285092 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285096 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285101 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285105 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285110 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285114 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285119 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285123 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285128 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285133 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285137 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285142 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285147 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285151 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285157 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285162 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:24:18.293609 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285167 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285171 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285175 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285180 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285186 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285192 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285197 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285202 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285207 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285212 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285217 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285240 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285245 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285250 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285255 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285260 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285265 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285270 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:24:18.294165 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:18.285274 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.285289 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.285514 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.288745 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.290268 2562 server.go:1019] "Starting client certificate rotation" Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.290378 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:24:18.294977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.290421 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:24:18.295355 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.295076 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:24:18.299409 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.299381 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:24:18.310232 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.310212 2562 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:24:18.315072 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.315055 2562 log.go:25] "Validated CRI v1 image API" Jul 06 20:24:18.317508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.317492 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:24:18.322015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.321977 2562 fs.go:135] Filesystem UUIDs: map[1d7f795c-5a16-4880-9b9a-b11ddf807146:/dev/nvme0n1p3 35a64b8e-3639-4705-8ea1-39fad27b608a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 20:24:18.322085 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.322014 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:24:18.329763 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.329591 2562 manager.go:216] Machine: {Timestamp:2026-07-06 20:24:18.328142445 +0000 UTC m=+0.869198875 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098992 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261225e71ad1645e6e19891fb7e101 SystemUUID:ec261225-e71a-d164-5e6e-19891fb7e101 BootID:0395ea01-857b-4443-a054-8b9e901a7374 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:5d:28:08:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:5d:28:08:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:8c:b4:e3:d7:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:24:18.329763 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.329752 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:24:18.329891 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.329851 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:24:18.332944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.332916 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:24:18.333109 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.332957 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:24:18.333760 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.333748 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:24:18.333793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.333763 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:24:18.333793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.333771 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:24:18.333793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.333788 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:24:18.335402 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.335390 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:24:18.338394 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.338376 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:24:18.338934 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.338922 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:24:18.339064 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339054 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:24:18.339095 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339087 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:24:18.339124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339098 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:24:18.339124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339115 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:24:18.339124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339124 2562 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:24:18.339252 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339136 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:24:18.339525 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339512 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:24:18.339562 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.339534 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:24:18.342007 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.341991 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:24:18.344295 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.344281 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:24:18.344355 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.344313 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:24:18.345711 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345695 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:24:18.345761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345721 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:24:18.345761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345732 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:24:18.345761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345742 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:24:18.345761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345757 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345765 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345785 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345810 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345818 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345824 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345837 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:24:18.345876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.345844 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:24:18.346614 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.346604 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:24:18.346655 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.346615 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:24:18.350338 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.350313 2562 server.go:1268] "Started kubelet" Jul 06 20:24:18.350496 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.350417 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:24:18.350547 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.350530 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:24:18.350963 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.350715 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:24:18.351370 ip-10-0-134-98 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:24:18.352008 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.351644 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:24:18.352008 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.351665 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:24:18.352008 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.351697 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:24:18.352436 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.352417 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:24:18.353798 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.353774 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:24:18.358573 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.358542 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:24:18.358793 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.357774 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12dc3fb7cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.350274509 +0000 UTC m=+0.891330939,LastTimestamp:2026-07-06 20:24:18.350274509 +0000 UTC m=+0.891330939,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.359907 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.359894 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:24:18.359959 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.359949 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:24:18.360127 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.360105 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.360635 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.360612 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:24:18.360635 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.360635 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:24:18.360793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.360775 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:24:18.360891 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.360815 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:24:18.360953 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.360924 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:24:18.361357 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361339 2562 factory.go:55] Registering systemd factory Jul 06 20:24:18.361357 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361359 2562 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:24:18.361585 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361568 2562 factory.go:153] Registering CRI-O factory Jul 06 20:24:18.361585 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361589 2562 factory.go:223] Registration of the crio container factory successfully Jul 06 20:24:18.361718 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361643 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:24:18.361718 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361673 2562 factory.go:103] Registering Raw factory Jul 06 20:24:18.361718 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.361691 2562 manager.go:1196] Started watching for new ooms in manager Jul 06 20:24:18.362479 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.362464 2562 manager.go:349] Starting recovery of all containers Jul 06 20:24:18.373660 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.373620 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 20:24:18.374775 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.374685 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 20:24:18.378293 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.378107 2562 manager.go:354] Recovery completed Jul 06 20:24:18.383068 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.383053 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.388279 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.388259 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.388350 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.388290 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.388350 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.388304 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.388980 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.388963 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:24:18.388980 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.388978 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:24:18.389109 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.389001 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:24:18.389109 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.389016 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.389808 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.389793 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.389864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.389819 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.389864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.389830 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.390412 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.390397 2562 policy_none.go:49] "None policy: Start" Jul 06 20:24:18.390470 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.390419 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:24:18.390470 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.390435 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:24:18.391520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.391507 2562 policy_none.go:47] "Start" Jul 06 20:24:18.398178 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.398098 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de839586 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388276614 +0000 UTC m=+0.929333044,LastTimestamp:2026-07-06 20:24:18.388276614 +0000 UTC m=+0.929333044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.412617 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.412531 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de83e013 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388295699 +0000 UTC m=+0.929352130,LastTimestamp:2026-07-06 20:24:18.388295699 +0000 UTC m=+0.929352130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.421529 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.421435 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de8418f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388310263 +0000 UTC m=+0.929366697,LastTimestamp:2026-07-06 20:24:18.388310263 +0000 UTC m=+0.929366697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.425667 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.425588 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-98.ec2.internal.18bfcd12de839586\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de839586 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388276614 +0000 UTC m=+0.929333044,LastTimestamp:2026-07-06 20:24:18.389806239 +0000 UTC m=+0.930862670,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.431940 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.431917 2562 manager.go:337] "Starting Device Plugin manager" Jul 06 20:24:18.432020 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.431954 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:24:18.432020 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.431966 2562 server.go:87] "Starting device plugin registration server" Jul 06 20:24:18.432851 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.432829 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:24:18.432946 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.432845 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:24:18.433021 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.433009 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:24:18.433159 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.433134 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:24:18.433159 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.433151 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:24:18.434679 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.434602 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:24:18.434679 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.434664 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.438398 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.438326 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-98.ec2.internal.18bfcd12de83e013\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de83e013 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388295699 +0000 UTC m=+0.929352130,LastTimestamp:2026-07-06 20:24:18.389824157 +0000 UTC m=+0.930880588,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.459499 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.459415 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-98.ec2.internal.18bfcd12de8418f7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12de8418f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.388310263 +0000 UTC m=+0.929366697,LastTimestamp:2026-07-06 20:24:18.389834365 +0000 UTC m=+0.930890796,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.472951 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.472874 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-98.ec2.internal.18bfcd12e14cc971 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-98.ec2.internal,UID:ip-10-0-134-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-98.ec2.internal,},FirstTimestamp:2026-07-06 20:24:18.435017073 +0000 UTC m=+0.976073503,LastTimestamp:2026-07-06 20:24:18.435017073 +0000 UTC m=+0.976073503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-98.ec2.internal,}" Jul 06 20:24:18.487940 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.487911 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wsbdw" Jul 06 20:24:18.512088 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.512011 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wsbdw" Jul 06 20:24:18.533488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.533462 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.534548 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.534529 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.534645 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.534563 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.534645 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.534574 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.534645 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.534619 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.550926 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.550883 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:24:18.552136 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.552114 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:24:18.552205 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.552155 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:24:18.552205 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.552178 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:24:18.552310 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.552212 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:24:18.555353 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.555331 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:24:18.555508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.555492 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.555561 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.555517 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-98.ec2.internal\": node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.653365 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.653314 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal"] Jul 06 20:24:18.653579 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.653430 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.654558 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.654539 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.654627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.654575 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.654627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.654590 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.656819 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.656790 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.656934 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.656857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.656934 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.656892 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.657977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.657959 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.658071 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.657991 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.658071 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.658002 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.658071 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.657959 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.658071 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.658069 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.658249 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.658080 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.659244 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.659220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.659288 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.659251 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:18.659949 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.659931 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:18.660020 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.659966 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:18.660020 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.659983 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:18.662837 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.662820 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.668182 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.668149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100977fbe6cbefbba384bf08f4769a9b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-98.ec2.internal\" (UID: \"100977fbe6cbefbba384bf08f4769a9b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.668259 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.668191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.668259 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.668211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.683197 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.683173 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-98.ec2.internal\" not found" node="ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.688196 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.688178 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-98.ec2.internal\" not found" node="ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.763175 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.763076 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.769103 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.769172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.769172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100977fbe6cbefbba384bf08f4769a9b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-98.ec2.internal\" (UID: \"100977fbe6cbefbba384bf08f4769a9b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.769300 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769194 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100977fbe6cbefbba384bf08f4769a9b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-98.ec2.internal\" (UID: \"100977fbe6cbefbba384bf08f4769a9b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.769300 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769223 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.769300 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.769224 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e111373156de52f0823c76ca1df26bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal\" (UID: \"6e111373156de52f0823c76ca1df26bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.864218 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.864182 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.964341 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:18.964296 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:18.987117 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.987079 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:18.992480 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:18.992455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:19.065556 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.065517 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.166309 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.166282 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.267096 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.267056 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.289801 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.289768 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:24:19.289949 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.289930 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:24:19.367635 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.367544 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.431324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.431293 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:24:19.434136 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.434112 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:24:19.434333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.434152 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:24:19.434333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.434169 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:24:19.468212 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.468179 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.514584 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.514534 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:19:19 +0000 UTC" deadline="2028-01-04 15:23:16.295730337 +0000 UTC" Jul 06 20:24:19.514584 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.514574 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13122h58m56.781159811s" Jul 06 20:24:19.552949 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:19.552893 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e111373156de52f0823c76ca1df26bb.slice/crio-1bfd693dea071be029400ea16e0b88f98bf253ec89723f357d2de56f181c98c9 WatchSource:0}: Error finding container 1bfd693dea071be029400ea16e0b88f98bf253ec89723f357d2de56f181c98c9: Status 404 returned error can't find the container with id 1bfd693dea071be029400ea16e0b88f98bf253ec89723f357d2de56f181c98c9 Jul 06 20:24:19.553595 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:19.553571 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100977fbe6cbefbba384bf08f4769a9b.slice/crio-0aa1f81f078108528404cc9df6a2a1989b5d337cb3099d0cc39de8ef73414390 WatchSource:0}: Error finding container 0aa1f81f078108528404cc9df6a2a1989b5d337cb3099d0cc39de8ef73414390: Status 404 returned error can't find the container with id 0aa1f81f078108528404cc9df6a2a1989b5d337cb3099d0cc39de8ef73414390 Jul 06 20:24:19.558694 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.558673 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:24:19.569134 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.569110 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.669313 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:19.669195 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-98.ec2.internal\" not found" Jul 06 20:24:19.732778 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.732745 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:24:19.757452 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.757427 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:24:19.760717 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.760696 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" Jul 06 20:24:19.783781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.783754 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:24:19.785558 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.785541 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" Jul 06 20:24:19.808590 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.808564 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:24:19.852163 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:19.852111 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:24:20.342325 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.342272 2562 apiserver.go:52] "Watching apiserver" Jul 06 20:24:20.351807 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.351564 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:24:20.352951 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.352749 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-zh9rz","openshift-multus/network-metrics-daemon-8jd6j","openshift-ovn-kubernetes/ovnkube-node-k5qb8","kube-system/konnectivity-agent-k4xpp","kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr","openshift-cluster-node-tuning-operator/tuned-jpnpp","openshift-image-registry/node-ca-h8rnd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal","openshift-network-diagnostics/network-check-target-f74sm","openshift-network-operator/iptables-alerter-qw2p2","kube-system/global-pull-secret-syncer-z2l27","openshift-dns/node-resolver-8np8r","openshift-multus/multus-additional-cni-plugins-kxzmh"] Jul 06 20:24:20.354861 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.354822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.356189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.356150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.357629 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.357413 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.357847 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.358051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.358756 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2h2t\"" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.358901 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.359025 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vhrl\"" Jul 06 20:24:20.359150 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.359037 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.359454 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.359214 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:24:20.359558 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.359537 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.361301 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.361774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-97jcw\"" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.361951 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.362362 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.362403 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.362445 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.362475 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:24:20.362990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.362749 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.363825 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.363799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:24:20.364452 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.364426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:24:20.364452 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.364448 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d4l4w\"" Jul 06 20:24:20.365363 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365341 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:24:20.365472 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365370 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-65zvg\"" Jul 06 20:24:20.365472 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365451 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:24:20.365586 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365472 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.365586 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365376 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:24:20.365691 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.365632 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.367419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.367399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.368071 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.368047 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:24:20.369883 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.369868 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.370356 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.370339 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hjq2z\"" Jul 06 20:24:20.370781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.370765 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:24:20.371111 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.371096 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.371486 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.371465 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.372673 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.372646 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.373662 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.373535 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:24:20.374640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374196 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n8h2g\"" Jul 06 20:24:20.374640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.374640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374315 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:24:20.374640 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.374347 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:20.374640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374506 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.374992 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374649 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.374992 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.374947 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:24:20.375372 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.375350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzs7x\"" Jul 06 20:24:20.375466 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.375354 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:24:20.376135 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.376118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:20.376233 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.376198 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:20.379201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-device-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.379323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-ovn\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj6r\" (UniqueName: \"kubernetes.io/projected/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-kube-api-access-qpj6r\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmp6g\" (UniqueName: \"kubernetes.io/projected/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kube-api-access-wmp6g\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-sys\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-slash\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-netns\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-var-lib-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-env-overrides\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/babd5803-94f9-49a1-8814-a60283349a51-agent-certs\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-run\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-os-release\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-tmp\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-script-lib\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-kubernetes\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-host-slash\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-var-lib-kubelet\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9752\" (UniqueName: \"kubernetes.io/projected/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-kube-api-access-j9752\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.379797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-etc-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-etc-tuned\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379878 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-host\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379901 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-kubelet\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-cnibin\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.379999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-socket-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-bin\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-config\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovn-node-metrics-cert\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/babd5803-94f9-49a1-8814-a60283349a51-konnectivity-ca\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380250 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2cf\" (UniqueName: \"kubernetes.io/projected/95a80467-6e7b-4005-b934-387dd6780512-kube-api-access-4b2cf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.380417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-systemd\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-lib-modules\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqnh\" (UniqueName: \"kubernetes.io/projected/d7ae9a77-fee5-4386-9614-29fac591c309-kube-api-access-8zqnh\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-sys-fs\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-modprobe-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-systemd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380448 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysconfig\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-systemd-units\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-netd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-registration-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-node-log\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-log-socket\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-iptables-alerter-script\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.381310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.380732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-conf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.382415 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.382392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.384108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.384088 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.384221 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.384162 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:20.385140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.385118 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sgzvl\"" Jul 06 20:24:20.385400 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.385382 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:24:20.398167 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.398102 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q9tgh" Jul 06 20:24:20.407695 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.407669 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q9tgh" Jul 06 20:24:20.462886 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.462847 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:24:20.481761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.481722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cnibin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.481761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.481762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-kubelet\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.482043 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.481796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-kubernetes\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.482365 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-host-slash\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.482438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-k8s-cni-cncf-io\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.482438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-var-lib-kubelet\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.482517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9752\" (UniqueName: \"kubernetes.io/projected/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-kube-api-access-j9752\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.482517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-os-release\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.482517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-etc-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.482669 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482563 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-kubernetes\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.482669 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482613 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-var-lib-kubelet\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.482832 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.482788 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-host-slash\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.483209 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-etc-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.483302 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-system-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.483302 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-netns\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.483402 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-hostroot\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.483402 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7083172-62a8-4506-b77e-7112ccde6e98-hosts-file\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.483402 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.483543 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-etc-tuned\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.483543 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483537 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.483643 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/212a155d-c62b-42a3-bbeb-d230b050d072-serviceca\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.483643 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-host\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.483643 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-kubelet\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.483846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-cnibin\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.483846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.483846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.483846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483853 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-socket-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-bin\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-config\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.483984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovn-node-metrics-cert\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.484033 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.484015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/babd5803-94f9-49a1-8814-a60283349a51-konnectivity-ca\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.484864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-multus\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.484908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-conf-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.484947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-daemon-config\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.484986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2cf\" (UniqueName: \"kubernetes.io/projected/95a80467-6e7b-4005-b934-387dd6780512-kube-api-access-4b2cf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485053 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-multus-certs\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-systemd\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-lib-modules\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqnh\" (UniqueName: \"kubernetes.io/projected/d7ae9a77-fee5-4386-9614-29fac591c309-kube-api-access-8zqnh\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-sys-fs\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-modprobe-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-systemd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cni-binary-copy\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-socket-dir-parent\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysconfig\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.485996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-systemd-units\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhj7f\" (UniqueName: \"kubernetes.io/projected/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-kube-api-access-qhj7f\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbxc\" (UniqueName: \"kubernetes.io/projected/f7083172-62a8-4506-b77e-7112ccde6e98-kube-api-access-vvbxc\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485557 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-bin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/212a155d-c62b-42a3-bbeb-d230b050d072-host\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-netd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485727 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-kubelet\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485821 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-systemd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485914 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-netd\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.485917 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-lib-modules\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486309 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/babd5803-94f9-49a1-8814-a60283349a51-konnectivity-ca\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486333 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486544 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-cnibin\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486584 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-host\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.486877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486669 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysconfig\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486693 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-systemd-units\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.486756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7083172-62a8-4506-b77e-7112ccde6e98-tmp-dir\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487055 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-socket-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487112 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487183 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-systemd\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487199 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-cni-bin\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487431 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-sys-fs\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487465 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-config\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-dbus\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487510 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.487733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.487537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.488333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488198 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.488333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-registration-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.488438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-node-log\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-log-socket\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488501 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-iptables-alerter-script\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.488501 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488496 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-log-socket\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-registration-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-conf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488577 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-node-log\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-device-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.488589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488578 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovn-node-metrics-cert\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwcp\" (UniqueName: \"kubernetes.io/projected/212a155d-c62b-42a3-bbeb-d230b050d072-kube-api-access-8rwcp\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488650 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-ovn\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488704 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-sysctl-conf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488724 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7ae9a77-fee5-4386-9614-29fac591c309-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488793 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-device-dir\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj6r\" (UniqueName: \"kubernetes.io/projected/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-kube-api-access-qpj6r\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488828 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-etc-kubernetes\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.488989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.488963 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6gg\" (UniqueName: \"kubernetes.io/projected/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-kube-api-access-ll6gg\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp6g\" (UniqueName: \"kubernetes.io/projected/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kube-api-access-wmp6g\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489097 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-etc-modprobe-d\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489130 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-iptables-alerter-script\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-sys\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489189 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-run-ovn\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489189 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-sys\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.489342 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-slash\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489400 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-slash\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-etc-tuned\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-netns\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-var-lib-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-var-lib-openvswitch\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-env-overrides\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489540 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-host-run-netns\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/babd5803-94f9-49a1-8814-a60283349a51-agent-certs\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-kubelet-config\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.489738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-run\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-os-release\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-tmp\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-script-lib\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489931 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-env-overrides\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.489996 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.490038 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95a80467-6e7b-4005-b934-387dd6780512-run\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.490141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.490086 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7ae9a77-fee5-4386-9614-29fac591c309-os-release\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.490562 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.490466 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-ovnkube-script-lib\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.492377 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.492348 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/babd5803-94f9-49a1-8814-a60283349a51-agent-certs\") pod \"konnectivity-agent-k4xpp\" (UID: \"babd5803-94f9-49a1-8814-a60283349a51\") " pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.493429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.493405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9752\" (UniqueName: \"kubernetes.io/projected/65b87a47-189a-4a16-b7b8-a91d0e22c7e2-kube-api-access-j9752\") pod \"iptables-alerter-qw2p2\" (UID: \"65b87a47-189a-4a16-b7b8-a91d0e22c7e2\") " pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.493564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.493541 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95a80467-6e7b-4005-b934-387dd6780512-tmp\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.498697 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.498672 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp6g\" (UniqueName: \"kubernetes.io/projected/07a3a2dc-8fe6-465c-9d72-df68eafafa8e-kube-api-access-wmp6g\") pod \"aws-ebs-csi-driver-node-qckqr\" (UID: \"07a3a2dc-8fe6-465c-9d72-df68eafafa8e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.498799 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.498779 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj6r\" (UniqueName: \"kubernetes.io/projected/5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb-kube-api-access-qpj6r\") pod \"ovnkube-node-k5qb8\" (UID: \"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.500370 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.500347 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2cf\" (UniqueName: \"kubernetes.io/projected/95a80467-6e7b-4005-b934-387dd6780512-kube-api-access-4b2cf\") pod \"tuned-jpnpp\" (UID: \"95a80467-6e7b-4005-b934-387dd6780512\") " pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.500479 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.500461 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqnh\" (UniqueName: \"kubernetes.io/projected/d7ae9a77-fee5-4386-9614-29fac591c309-kube-api-access-8zqnh\") pod \"multus-additional-cni-plugins-kxzmh\" (UID: \"d7ae9a77-fee5-4386-9614-29fac591c309\") " pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.560579 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.560513 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" event={"ID":"100977fbe6cbefbba384bf08f4769a9b","Type":"ContainerStarted","Data":"0aa1f81f078108528404cc9df6a2a1989b5d337cb3099d0cc39de8ef73414390"} Jul 06 20:24:20.561814 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.561786 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" event={"ID":"6e111373156de52f0823c76ca1df26bb","Type":"ContainerStarted","Data":"1bfd693dea071be029400ea16e0b88f98bf253ec89723f357d2de56f181c98c9"} Jul 06 20:24:20.590295 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cnibin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-kubelet\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-k8s-cni-cncf-io\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-os-release\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-system-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-netns\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-hostroot\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7083172-62a8-4506-b77e-7112ccde6e98-hosts-file\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/212a155d-c62b-42a3-bbeb-d230b050d072-serviceca\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-multus\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590579 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-os-release\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-conf-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-daemon-config\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590652 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-conf-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590718 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-system-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590777 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cnibin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590817 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-kubelet\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.590869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590858 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-k8s-cni-cncf-io\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590869 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-cni-dir\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-multus-certs\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590923 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-multus\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590943 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-hostroot\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590955 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cni-binary-copy\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590957 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-netns\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590990 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-run-multus-certs\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.590995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-socket-dir-parent\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhj7f\" (UniqueName: \"kubernetes.io/projected/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-kube-api-access-qhj7f\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591043 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7083172-62a8-4506-b77e-7112ccde6e98-hosts-file\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591070 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-socket-dir-parent\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbxc\" (UniqueName: \"kubernetes.io/projected/f7083172-62a8-4506-b77e-7112ccde6e98-kube-api-access-vvbxc\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-bin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/212a155d-c62b-42a3-bbeb-d230b050d072-host\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.591461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7083172-62a8-4506-b77e-7112ccde6e98-tmp-dir\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591243 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-host-var-lib-cni-bin\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-dbus\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwcp\" (UniqueName: \"kubernetes.io/projected/212a155d-c62b-42a3-bbeb-d230b050d072-kube-api-access-8rwcp\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-etc-kubernetes\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6gg\" (UniqueName: \"kubernetes.io/projected/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-kube-api-access-ll6gg\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-kubelet-config\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591436 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/212a155d-c62b-42a3-bbeb-d230b050d072-serviceca\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591470 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-dbus\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591474 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f96f3b-f7be-4967-a67a-6b3952922001-kubelet-config\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.591334 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-etc-kubernetes\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.591570 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:21.091548247 +0000 UTC m=+3.632604668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591607 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-cni-binary-copy\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591634 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-multus-daemon-config\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591672 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/212a155d-c62b-42a3-bbeb-d230b050d072-host\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.591685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7083172-62a8-4506-b77e-7112ccde6e98-tmp-dir\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.592376 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.591708 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:20.593143 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.591773 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:21.09175561 +0000 UTC m=+3.632812044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:20.602423 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.602397 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbxc\" (UniqueName: \"kubernetes.io/projected/f7083172-62a8-4506-b77e-7112ccde6e98-kube-api-access-vvbxc\") pod \"node-resolver-8np8r\" (UID: \"f7083172-62a8-4506-b77e-7112ccde6e98\") " pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.602678 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.602660 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:20.602765 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.602683 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:20.602765 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.602697 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:20.602886 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:20.602776 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:21.102756377 +0000 UTC m=+3.643812830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:20.606271 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.606224 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwcp\" (UniqueName: \"kubernetes.io/projected/212a155d-c62b-42a3-bbeb-d230b050d072-kube-api-access-8rwcp\") pod \"node-ca-h8rnd\" (UID: \"212a155d-c62b-42a3-bbeb-d230b050d072\") " pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.607886 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.607861 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhj7f\" (UniqueName: \"kubernetes.io/projected/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-kube-api-access-qhj7f\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:20.608098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.608075 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6gg\" (UniqueName: \"kubernetes.io/projected/9db1a4de-8cee-45bf-abe4-a0a72a3b7999-kube-api-access-ll6gg\") pod \"multus-zh9rz\" (UID: \"9db1a4de-8cee-45bf-abe4-a0a72a3b7999\") " pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.673363 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.673325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" Jul 06 20:24:20.683588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.683554 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qw2p2" Jul 06 20:24:20.684957 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:20.684928 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a80467_6e7b_4005_b934_387dd6780512.slice/crio-2b78d4c44496c4ed8d6c78cf9932f2af16c01521b8963b42247cb1ff9708c7ca WatchSource:0}: Error finding container 2b78d4c44496c4ed8d6c78cf9932f2af16c01521b8963b42247cb1ff9708c7ca: Status 404 returned error can't find the container with id 2b78d4c44496c4ed8d6c78cf9932f2af16c01521b8963b42247cb1ff9708c7ca Jul 06 20:24:20.692656 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:20.692631 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b87a47_189a_4a16_b7b8_a91d0e22c7e2.slice/crio-e3ec48a18ee452d93d2e32a2d6fd42606d53f22e1cca6dabc13286650f4cd185 WatchSource:0}: Error finding container e3ec48a18ee452d93d2e32a2d6fd42606d53f22e1cca6dabc13286650f4cd185: Status 404 returned error can't find the container with id e3ec48a18ee452d93d2e32a2d6fd42606d53f22e1cca6dabc13286650f4cd185 Jul 06 20:24:20.695893 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.695876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:20.700877 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.700857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:20.709345 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.709317 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" Jul 06 20:24:20.718308 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.718275 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" Jul 06 20:24:20.734442 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.734415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h8rnd" Jul 06 20:24:20.740990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.740966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8np8r" Jul 06 20:24:20.748774 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.748748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zh9rz" Jul 06 20:24:20.763210 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:20.763188 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:24:21.094809 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.094763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:21.095011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.094833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:21.095011 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.094937 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:21.095011 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.094974 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:21.095203 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.095025 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:22.095002753 +0000 UTC m=+4.636059175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:21.095203 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.095043 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:22.095034301 +0000 UTC m=+4.636090723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:21.196125 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.196079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:21.196336 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.196311 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:21.196388 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.196348 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:21.196388 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.196363 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:21.196467 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.196439 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:22.196417676 +0000 UTC m=+4.737474113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:21.412019 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.411893 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:19:20 +0000 UTC" deadline="2028-04-07 12:37:27.334771102 +0000 UTC" Jul 06 20:24:21.412019 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.411958 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15376h13m5.922817314s" Jul 06 20:24:21.553468 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.553432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:21.553650 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:21.553582 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:21.565063 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.565025 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qw2p2" event={"ID":"65b87a47-189a-4a16-b7b8-a91d0e22c7e2","Type":"ContainerStarted","Data":"e3ec48a18ee452d93d2e32a2d6fd42606d53f22e1cca6dabc13286650f4cd185"} Jul 06 20:24:21.566327 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:21.566294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" event={"ID":"95a80467-6e7b-4005-b934-387dd6780512","Type":"ContainerStarted","Data":"2b78d4c44496c4ed8d6c78cf9932f2af16c01521b8963b42247cb1ff9708c7ca"} Jul 06 20:24:21.769366 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.769325 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212a155d_c62b_42a3_bbeb_d230b050d072.slice/crio-4036acf301017d4b4691a4a4627ed7f41b2b7217d5687ad272c881fc61052180 WatchSource:0}: Error finding container 4036acf301017d4b4691a4a4627ed7f41b2b7217d5687ad272c881fc61052180: Status 404 returned error can't find the container with id 4036acf301017d4b4691a4a4627ed7f41b2b7217d5687ad272c881fc61052180 Jul 06 20:24:21.770652 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.770624 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a3a2dc_8fe6_465c_9d72_df68eafafa8e.slice/crio-dfe8e7f59ae01db12a98565a7b74c31b96111d7b57a25fbe3f84d2dbc1c79f59 WatchSource:0}: Error finding container dfe8e7f59ae01db12a98565a7b74c31b96111d7b57a25fbe3f84d2dbc1c79f59: Status 404 returned error can't find the container with id dfe8e7f59ae01db12a98565a7b74c31b96111d7b57a25fbe3f84d2dbc1c79f59 Jul 06 20:24:21.771484 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.771454 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db1a4de_8cee_45bf_abe4_a0a72a3b7999.slice/crio-059f039813fef84fb3268c2310064948d6c827cf242d17470309a6ab901f430e WatchSource:0}: Error finding container 059f039813fef84fb3268c2310064948d6c827cf242d17470309a6ab901f430e: Status 404 returned error can't find the container with id 059f039813fef84fb3268c2310064948d6c827cf242d17470309a6ab901f430e Jul 06 20:24:21.775186 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.775140 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ae9a77_fee5_4386_9614_29fac591c309.slice/crio-f40d72d1686117b168f44c11310ad65352827e3842f1ef779fd6a34708616a97 WatchSource:0}: Error finding container f40d72d1686117b168f44c11310ad65352827e3842f1ef779fd6a34708616a97: Status 404 returned error can't find the container with id f40d72d1686117b168f44c11310ad65352827e3842f1ef779fd6a34708616a97 Jul 06 20:24:21.776089 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.776039 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7083172_62a8_4506_b77e_7112ccde6e98.slice/crio-100f71e47c093be9bea15b72832fb31ae7351ecaaca2bfded1a9e0533b0c7ecb WatchSource:0}: Error finding container 100f71e47c093be9bea15b72832fb31ae7351ecaaca2bfded1a9e0533b0c7ecb: Status 404 returned error can't find the container with id 100f71e47c093be9bea15b72832fb31ae7351ecaaca2bfded1a9e0533b0c7ecb Jul 06 20:24:21.777667 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:21.777647 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1ad4de_44c2_49cb_84a2_1fb6ff2697bb.slice/crio-62b36ee05873722017d2c61d3b23d31435726bd8cbe3f6b23fdc89a102dd1179 WatchSource:0}: Error finding container 62b36ee05873722017d2c61d3b23d31435726bd8cbe3f6b23fdc89a102dd1179: Status 404 returned error can't find the container with id 62b36ee05873722017d2c61d3b23d31435726bd8cbe3f6b23fdc89a102dd1179 Jul 06 20:24:22.104847 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.104586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:22.104986 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.104878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:22.104986 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.104737 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:22.104986 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.104975 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:24.104952192 +0000 UTC m=+6.646008610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:22.105163 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.105024 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:22.105163 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.105090 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:24.105064035 +0000 UTC m=+6.646120459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:22.206914 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.206254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:22.206914 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.206443 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:22.206914 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.206465 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:22.206914 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.206480 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:22.206914 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.206547 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:24.206527143 +0000 UTC m=+6.747583601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:22.413193 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.413128 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:19:20 +0000 UTC" deadline="2028-01-02 10:21:10.291657475 +0000 UTC" Jul 06 20:24:22.413193 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.413190 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13069h56m47.878472284s" Jul 06 20:24:22.556587 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.555807 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:22.556587 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.555940 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:22.556587 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.556381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:22.556587 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:22.556484 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:22.583964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.583476 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e111373156de52f0823c76ca1df26bb" containerID="c5c76bf7eb71446e080d73389efe96e80eb0c65f90236ba2f9d49e27aa6bd267" exitCode=0 Jul 06 20:24:22.583964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.583575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" event={"ID":"6e111373156de52f0823c76ca1df26bb","Type":"ContainerDied","Data":"c5c76bf7eb71446e080d73389efe96e80eb0c65f90236ba2f9d49e27aa6bd267"} Jul 06 20:24:22.591042 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.590965 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4xpp" event={"ID":"babd5803-94f9-49a1-8814-a60283349a51","Type":"ContainerStarted","Data":"32eb2170e36b9a4be98998b20664e6fca7dc9fc4d4ef7789794bbaaab64c3e61"} Jul 06 20:24:22.601159 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.601123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"62b36ee05873722017d2c61d3b23d31435726bd8cbe3f6b23fdc89a102dd1179"} Jul 06 20:24:22.609434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.609398 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8np8r" event={"ID":"f7083172-62a8-4506-b77e-7112ccde6e98","Type":"ContainerStarted","Data":"100f71e47c093be9bea15b72832fb31ae7351ecaaca2bfded1a9e0533b0c7ecb"} Jul 06 20:24:22.627123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.627077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8rnd" event={"ID":"212a155d-c62b-42a3-bbeb-d230b050d072","Type":"ContainerStarted","Data":"4036acf301017d4b4691a4a4627ed7f41b2b7217d5687ad272c881fc61052180"} Jul 06 20:24:22.637789 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.637740 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" event={"ID":"100977fbe6cbefbba384bf08f4769a9b","Type":"ContainerStarted","Data":"74dbafc623d87d086274b44b6ee58ec99d351dffea53509861c9a3d80f84894e"} Jul 06 20:24:22.651244 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.651193 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerStarted","Data":"f40d72d1686117b168f44c11310ad65352827e3842f1ef779fd6a34708616a97"} Jul 06 20:24:22.675729 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.675570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zh9rz" event={"ID":"9db1a4de-8cee-45bf-abe4-a0a72a3b7999","Type":"ContainerStarted","Data":"059f039813fef84fb3268c2310064948d6c827cf242d17470309a6ab901f430e"} Jul 06 20:24:22.693201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:22.693155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" event={"ID":"07a3a2dc-8fe6-465c-9d72-df68eafafa8e","Type":"ContainerStarted","Data":"dfe8e7f59ae01db12a98565a7b74c31b96111d7b57a25fbe3f84d2dbc1c79f59"} Jul 06 20:24:23.553392 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:23.553352 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:23.554120 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:23.553535 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:23.716346 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:23.716300 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" event={"ID":"6e111373156de52f0823c76ca1df26bb","Type":"ContainerStarted","Data":"a37f00b0adfe60fab0ea0ccef3616ff49341f4024be71e9c41eb9ebb885df0d9"} Jul 06 20:24:23.736827 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:23.736759 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-98.ec2.internal" podStartSLOduration=3.736734189 podStartE2EDuration="3.736734189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:22.662186835 +0000 UTC m=+5.203243275" watchObservedRunningTime="2026-07-06 20:24:23.736734189 +0000 UTC m=+6.277790631" Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:24.125668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:24.125729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.125928 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.125997 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:28.125976346 +0000 UTC m=+10.667032781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.126296 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:24.126386 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.126391 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:28.126363722 +0000 UTC m=+10.667420152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:24.226852 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:24.226795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:24.227044 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.227002 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:24.227044 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.227026 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:24.227044 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.227039 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:24.227243 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.227107 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:28.227082445 +0000 UTC m=+10.768138877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:24.553398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:24.553306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:24.553570 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.553465 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:24.553996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:24.553974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:24.554120 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:24.554098 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:25.553058 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:25.553015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:25.553291 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:25.553196 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:26.553242 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:26.553180 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:26.553777 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:26.553350 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:26.553939 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:26.553916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:26.554068 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:26.554043 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:27.553885 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:27.553369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:27.553885 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:27.553519 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:28.159331 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:28.159286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:28.159520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:28.159341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:28.159520 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.159439 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:28.159520 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.159515 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:36.15949458 +0000 UTC m=+18.700551015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:28.159700 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.159447 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:28.159700 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.159565 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:36.159552775 +0000 UTC m=+18.700609193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:28.260143 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:28.260092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:28.260344 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.260267 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:28.260344 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.260288 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:28.260344 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.260301 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:28.260505 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.260362 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:36.260343207 +0000 UTC m=+18.801399623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:28.553879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:28.553220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:28.553879 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.553323 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:28.553879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:28.553695 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:28.553879 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:28.553775 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:29.553233 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:29.552692 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:29.553233 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:29.552838 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:30.553283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:30.553208 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:30.553283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:30.553254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:30.553876 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:30.553355 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:30.553876 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:30.553831 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:31.552904 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:31.552871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:31.553078 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:31.552988 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:32.552947 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:32.552913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:32.553349 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:32.552918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:32.553349 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:32.553058 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:32.553349 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:32.553101 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:33.552738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:33.552697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:33.552914 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:33.552853 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:34.552608 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:34.552552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:34.553092 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:34.552692 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:34.553092 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:34.552744 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:34.553092 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:34.552849 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:35.553261 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:35.553226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:35.553777 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:35.553361 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:36.224205 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:36.224162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:36.224348 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:36.224218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:36.224348 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.224259 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:36.224348 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.224293 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:36.224348 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.224333 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:52.224309549 +0000 UTC m=+34.765365970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:36.224348 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.224350 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:24:52.224341452 +0000 UTC m=+34.765397873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:36.325072 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:36.325036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:36.325231 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.325188 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:36.325231 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.325204 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:36.325231 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.325214 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:36.325364 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.325261 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:24:52.325248321 +0000 UTC m=+34.866304736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:36.552906 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:36.552823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:36.552906 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:36.552864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:36.553121 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.552974 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:36.553121 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:36.553053 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:37.553124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:37.553081 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:37.553539 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:37.553252 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:38.553321 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:38.553288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:38.553758 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:38.553380 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:38.553758 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:38.553485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:38.553758 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:38.553595 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:39.552856 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:39.552816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:39.553048 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:39.552941 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:40.552604 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:40.552563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:40.553283 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:40.552688 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:40.553283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:40.552753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:40.553283 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:40.552870 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:41.553606 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.553231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:41.554187 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:41.553731 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:41.764202 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.764164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4xpp" event={"ID":"babd5803-94f9-49a1-8814-a60283349a51","Type":"ContainerStarted","Data":"cb4c5c6e9fb8a549aa34b8cde54b37d2fdfd3125cd57658ab4dc42f30c3a531f"} Jul 06 20:24:41.767308 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767281 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"c6bd2641e9692fcc3fb2230bbcdf44a94ca7e6f32b2c7ba65447fdd0f389adb7"} Jul 06 20:24:41.767463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"5d07ff4ee6c6a1314180e8e998558be60ddad852ef4a4c31a1ecda8291c0087a"} Jul 06 20:24:41.767463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"ce6c8390d5f3574393bc7b7d33f052454f257060b3956123b7f5cf3974de058d"} Jul 06 20:24:41.767463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"7281f365df378578ee01026afe47b9c5fb1448b0adbb22866501427dbf4bb5e3"} Jul 06 20:24:41.767612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767495 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"153b046cab32e03011d0857957a5fe4527bd247c4de0c098a575bd52fe2424a2"} Jul 06 20:24:41.767612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.767526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"59dc5bd29280f2b5dd586db047191f2c5775fe90c09889fe04ba7b53ab826de7"} Jul 06 20:24:41.768720 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.768694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8np8r" event={"ID":"f7083172-62a8-4506-b77e-7112ccde6e98","Type":"ContainerStarted","Data":"f5eab7c58cea837d87edf2b2d17cf3ba05900657b18d124bd9cc8d39deb09202"} Jul 06 20:24:41.770100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.770077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h8rnd" event={"ID":"212a155d-c62b-42a3-bbeb-d230b050d072","Type":"ContainerStarted","Data":"a5268fcced88277b1dd522e101bcc22f4487c3ceedc85b8cd6ef904c5b91d71a"} Jul 06 20:24:41.771549 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.771528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" event={"ID":"95a80467-6e7b-4005-b934-387dd6780512","Type":"ContainerStarted","Data":"c1a38ffa60a881a0d642f87fe90d05fba27883042740c01d372126733da7c891"} Jul 06 20:24:41.773059 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.773035 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="6853f2d36c4f90f00ca6442f9dd6116f572a01be0aeb469c50ff2170b14c301f" exitCode=0 Jul 06 20:24:41.773170 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.773112 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"6853f2d36c4f90f00ca6442f9dd6116f572a01be0aeb469c50ff2170b14c301f"} Jul 06 20:24:41.774965 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.774898 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zh9rz" event={"ID":"9db1a4de-8cee-45bf-abe4-a0a72a3b7999","Type":"ContainerStarted","Data":"ef188bdba8b03f1a36b745c6c801dd709f663a8bdf04320223fb0b12fa74e1b9"} Jul 06 20:24:41.777011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.776992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" event={"ID":"07a3a2dc-8fe6-465c-9d72-df68eafafa8e","Type":"ContainerStarted","Data":"1993b96fdb0ac822e5795135056c7ee85a74a940cb466b805cbc65a837cfeecf"} Jul 06 20:24:41.810857 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.810803 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-98.ec2.internal" podStartSLOduration=21.810787288 podStartE2EDuration="21.810787288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:23.737528396 +0000 UTC m=+6.278584840" watchObservedRunningTime="2026-07-06 20:24:41.810787288 +0000 UTC m=+24.351843723" Jul 06 20:24:41.840667 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.840601 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k4xpp" podStartSLOduration=8.375874999 podStartE2EDuration="22.84057931s" totalImagesPullingTime="14.464704311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:41.810768216 +0000 UTC m=+24.351824652" watchObservedRunningTime="2026-07-06 20:24:41.84057931 +0000 UTC m=+24.381635749" Jul 06 20:24:41.862594 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.862548 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h8rnd" podStartSLOduration=4.026980111 podStartE2EDuration="22.862533592s" totalImagesPullingTime="18.835553481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:41.862191356 +0000 UTC m=+24.403247796" watchObservedRunningTime="2026-07-06 20:24:41.862533592 +0000 UTC m=+24.403590078" Jul 06 20:24:41.918632 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.918527 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jpnpp" podStartSLOduration=2.995847447 podStartE2EDuration="22.918510594s" totalImagesPullingTime="19.922663147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:41.918487269 +0000 UTC m=+24.459543707" watchObservedRunningTime="2026-07-06 20:24:41.918510594 +0000 UTC m=+24.459567071" Jul 06 20:24:41.918873 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.918843 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8np8r" podStartSLOduration=8.450745248 podStartE2EDuration="22.918834511s" totalImagesPullingTime="14.468089263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:41.889803346 +0000 UTC m=+24.430859784" watchObservedRunningTime="2026-07-06 20:24:41.918834511 +0000 UTC m=+24.459890947" Jul 06 20:24:41.945782 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:41.945731 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zh9rz" podStartSLOduration=4.058561124 podStartE2EDuration="22.945713966s" totalImagesPullingTime="18.887152842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:41.94569585 +0000 UTC m=+24.486752288" watchObservedRunningTime="2026-07-06 20:24:41.945713966 +0000 UTC m=+24.486770402" Jul 06 20:24:42.290384 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.290311 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:24:42.446991 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.446825 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:24:42.290324399Z","UUID":"a507f1a9-5bd3-42cc-ab14-1e725a8aa88a","Handler":null,"Name":"","Endpoint":""} Jul 06 20:24:42.448434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.448412 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:24:42.448434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.448451 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:24:42.553266 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.553236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:42.553433 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.553236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:42.553433 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:42.553360 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:42.553433 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:42.553401 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:42.780790 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.780702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" event={"ID":"07a3a2dc-8fe6-465c-9d72-df68eafafa8e","Type":"ContainerStarted","Data":"5f4076ef7c02dbf6dce36f93686c1f4d006376b8560206e7ce946221d8eccc1c"} Jul 06 20:24:42.782076 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:42.782045 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qw2p2" event={"ID":"65b87a47-189a-4a16-b7b8-a91d0e22c7e2","Type":"ContainerStarted","Data":"41aea8e833c6bac768d42a791496d587fc09df62c33f82b93174e9b32d62109e"} Jul 06 20:24:43.552955 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:43.552925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:43.553181 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:43.553054 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:43.789838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:43.789802 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"98a2aa98feca2978c067a1ca773f00563b0b91673098fbccc22bdad6f51eea77"} Jul 06 20:24:44.044038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.043955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:44.044579 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.044562 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:44.060618 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.060573 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qw2p2" podStartSLOduration=5.147314198 podStartE2EDuration="25.060558409s" totalImagesPullingTime="19.913244211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:42.8029332 +0000 UTC m=+25.343989636" watchObservedRunningTime="2026-07-06 20:24:44.060558409 +0000 UTC m=+26.601614845" Jul 06 20:24:44.553137 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.552897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:44.553315 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.552952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:44.553315 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:44.553252 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:44.553315 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:44.553272 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:44.793389 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.793352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" event={"ID":"07a3a2dc-8fe6-465c-9d72-df68eafafa8e","Type":"ContainerStarted","Data":"5fbbce6127344285fe4b059f4f30ac73edd2c587bfd86b7b267132675efcf415"} Jul 06 20:24:44.793848 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.793511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:44.794038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.794022 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4xpp" Jul 06 20:24:44.812365 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:44.812327 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qckqr" podStartSLOduration=4.261227816 podStartE2EDuration="25.812311668s" totalImagesPullingTime="21.551083852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:44.812198563 +0000 UTC m=+27.353255000" watchObservedRunningTime="2026-07-06 20:24:44.812311668 +0000 UTC m=+27.353368104" Jul 06 20:24:45.553067 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:45.553030 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:45.553263 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:45.553160 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:46.553415 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:46.553379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:46.554362 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:46.553522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:46.554362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:46.553578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:46.554362 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:46.554052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:46.802388 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:46.802347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" event={"ID":"5f1ad4de-44c2-49cb-84a2-1fb6ff2697bb","Type":"ContainerStarted","Data":"a29857a6eddbaf6d77c54d7e2f621b9c0754bfaaaeb9e74c3aaddc4021663fd5"} Jul 06 20:24:46.834070 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:46.834017 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" podStartSLOduration=8.921328575 podStartE2EDuration="27.834002033s" totalImagesPullingTime="18.912673458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:46.832833935 +0000 UTC m=+29.373890372" watchObservedRunningTime="2026-07-06 20:24:46.834002033 +0000 UTC m=+29.375058471" Jul 06 20:24:47.553419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.553380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:47.553591 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:47.553543 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:47.805758 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.805681 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:47.805919 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.805764 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:47.805919 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.805806 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:47.824789 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.824759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:47.825401 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:47.825378 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:24:48.204517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.204487 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jd6j"] Jul 06 20:24:48.204690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.204599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:48.204738 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:48.204690 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:48.206248 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.205979 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2l27"] Jul 06 20:24:48.206248 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.206084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:48.206248 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:48.206194 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:48.207215 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.207192 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f74sm"] Jul 06 20:24:48.207344 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.207290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:48.207404 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:48.207382 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:48.809370 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.809270 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="82d43e0ba3f2da307f604c5d6133c0e03ab7ba886d1a554f7ef5ecb5aee99de3" exitCode=0 Jul 06 20:24:48.809370 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:48.809354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"82d43e0ba3f2da307f604c5d6133c0e03ab7ba886d1a554f7ef5ecb5aee99de3"} Jul 06 20:24:49.552630 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:49.552594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:49.552785 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:49.552604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:49.552785 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:49.552744 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:49.552970 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:49.552859 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:49.814280 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:49.814242 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="cb5c76a5d43a995792078f32c0169216d700628b72d9e4e6b196b15fdcc4aa1c" exitCode=0 Jul 06 20:24:49.814675 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:49.814330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"cb5c76a5d43a995792078f32c0169216d700628b72d9e4e6b196b15fdcc4aa1c"} Jul 06 20:24:50.553199 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:50.553014 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:50.553366 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:50.553280 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:50.818963 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:50.818930 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="7938f96ec9cd4b3263ce48868c8a5cc733e3d1176286daa687f68b19ac2342df" exitCode=0 Jul 06 20:24:50.819467 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:50.818995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"7938f96ec9cd4b3263ce48868c8a5cc733e3d1176286daa687f68b19ac2342df"} Jul 06 20:24:51.552887 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:51.552843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:51.553042 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:51.552852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:51.553042 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:51.552952 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:51.553130 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:51.553044 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:52.244514 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:52.244478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:52.245060 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:52.244522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:52.245060 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.244639 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:52.245060 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.244642 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:52.245060 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.244702 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:25:24.244684606 +0000 UTC m=+66.785741021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:24:52.245060 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.244720 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret podName:49f96f3b-f7be-4967-a67a-6b3952922001 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:24.244711708 +0000 UTC m=+66.785768127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret") pod "global-pull-secret-syncer-z2l27" (UID: "49f96f3b-f7be-4967-a67a-6b3952922001") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:24:52.345525 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:52.345489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:52.345676 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.345654 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:24:52.345676 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.345676 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:24:52.345745 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.345687 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfkb9 for pod openshift-network-diagnostics/network-check-target-f74sm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:52.345745 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.345741 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9 podName:f8d1a285-7254-46cb-8c67-83df6a73c9fd nodeName:}" failed. No retries permitted until 2026-07-06 20:25:24.345726869 +0000 UTC m=+66.886783289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfkb9" (UniqueName: "kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9") pod "network-check-target-f74sm" (UID: "f8d1a285-7254-46cb-8c67-83df6a73c9fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:24:52.557349 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:52.557264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:52.557495 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:52.557367 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2l27" podUID="49f96f3b-f7be-4967-a67a-6b3952922001" Jul 06 20:24:53.552535 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:53.552496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:53.552971 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:53.552496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:53.552971 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:53.552663 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jd6j" podUID="1655b2df-cc8f-4633-87e9-1d5402e0ec7e" Jul 06 20:24:53.552971 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:53.552721 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f74sm" podUID="f8d1a285-7254-46cb-8c67-83df6a73c9fd" Jul 06 20:24:54.205104 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.205076 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-98.ec2.internal" event="NodeReady" Jul 06 20:24:54.205341 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.205261 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:24:54.259618 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.259586 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4glv9"] Jul 06 20:24:54.264684 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.264646 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h97f5"] Jul 06 20:24:54.264860 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.264788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.267078 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.267052 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvcv5\"" Jul 06 20:24:54.267218 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.267107 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:24:54.267450 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.267435 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:24:54.269404 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.269384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.271703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.271630 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:24:54.271703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.271644 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:24:54.271883 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.271645 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l5krs\"" Jul 06 20:24:54.271883 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.271719 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:24:54.280981 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.280042 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h97f5"] Jul 06 20:24:54.280981 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.280096 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4glv9"] Jul 06 20:24:54.358360 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/abb79635-08ac-4b40-93d2-f18435f585e8-config-volume\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.358560 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6195ed3c-3037-4c3e-85c5-f381fce16615-cert\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.358560 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sz2\" (UniqueName: \"kubernetes.io/projected/abb79635-08ac-4b40-93d2-f18435f585e8-kube-api-access-m4sz2\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.358560 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dfp\" (UniqueName: \"kubernetes.io/projected/6195ed3c-3037-4c3e-85c5-f381fce16615-kube-api-access-p9dfp\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.358560 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb79635-08ac-4b40-93d2-f18435f585e8-tmp-dir\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.358762 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.358661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.459490 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.459490 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459466 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/abb79635-08ac-4b40-93d2-f18435f585e8-config-volume\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6195ed3c-3037-4c3e-85c5-f381fce16615-cert\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:54.459523 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sz2\" (UniqueName: \"kubernetes.io/projected/abb79635-08ac-4b40-93d2-f18435f585e8-kube-api-access-m4sz2\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dfp\" (UniqueName: \"kubernetes.io/projected/6195ed3c-3037-4c3e-85c5-f381fce16615-kube-api-access-p9dfp\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:54.459592 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:54.959573588 +0000 UTC m=+37.500630002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:24:54.459712 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb79635-08ac-4b40-93d2-f18435f585e8-tmp-dir\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.459996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.459903 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb79635-08ac-4b40-93d2-f18435f585e8-tmp-dir\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.460119 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.460097 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/abb79635-08ac-4b40-93d2-f18435f585e8-config-volume\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.464350 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.464323 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6195ed3c-3037-4c3e-85c5-f381fce16615-cert\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.469630 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.469601 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sz2\" (UniqueName: \"kubernetes.io/projected/abb79635-08ac-4b40-93d2-f18435f585e8-kube-api-access-m4sz2\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.469806 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.469739 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dfp\" (UniqueName: \"kubernetes.io/projected/6195ed3c-3037-4c3e-85c5-f381fce16615-kube-api-access-p9dfp\") pod \"ingress-canary-h97f5\" (UID: \"6195ed3c-3037-4c3e-85c5-f381fce16615\") " pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.553451 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.553411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:24:54.556302 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.556268 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:24:54.583900 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.583865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h97f5" Jul 06 20:24:54.963314 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:54.963281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:54.963499 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:54.963406 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:24:54.963499 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:54.963467 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:55.963447312 +0000 UTC m=+38.504503728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:24:55.553257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.553220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:24:55.553437 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.553218 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:24:55.556764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.556692 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:24:55.556764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.556726 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:24:55.556764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.556739 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lrkdh\"" Jul 06 20:24:55.556764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.556763 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsjzg\"" Jul 06 20:24:55.557208 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.556777 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:24:55.745569 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.745369 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h97f5"] Jul 06 20:24:55.839353 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:24:55.839316 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6195ed3c_3037_4c3e_85c5_f381fce16615.slice/crio-ff523f2094c85f2d1ba859d3f70c068875458e83e448c382f5a8deb335c0b5f5 WatchSource:0}: Error finding container ff523f2094c85f2d1ba859d3f70c068875458e83e448c382f5a8deb335c0b5f5: Status 404 returned error can't find the container with id ff523f2094c85f2d1ba859d3f70c068875458e83e448c382f5a8deb335c0b5f5 Jul 06 20:24:55.971735 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:55.971703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:55.971938 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:55.971823 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:24:55.971938 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:55.971884 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:24:57.971869876 +0000 UTC m=+40.512926290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:24:56.835580 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:56.835543 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="df1b10dd2036712ae25387eaf2e92db2f548341c45256ab2f9d289384ea4c472" exitCode=0 Jul 06 20:24:56.836115 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:56.835583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"df1b10dd2036712ae25387eaf2e92db2f548341c45256ab2f9d289384ea4c472"} Jul 06 20:24:56.837562 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:56.837537 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h97f5" event={"ID":"6195ed3c-3037-4c3e-85c5-f381fce16615","Type":"ContainerStarted","Data":"ff523f2094c85f2d1ba859d3f70c068875458e83e448c382f5a8deb335c0b5f5"} Jul 06 20:24:57.843958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:57.843868 2562 generic.go:356] "Generic (PLEG): container finished" podID="d7ae9a77-fee5-4386-9614-29fac591c309" containerID="7eb77c702fb28b8c21470d6ec3cacbf243a55e43b03f06ff49de41f4b3fdf07c" exitCode=0 Jul 06 20:24:57.843958 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:57.843918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerDied","Data":"7eb77c702fb28b8c21470d6ec3cacbf243a55e43b03f06ff49de41f4b3fdf07c"} Jul 06 20:24:57.985161 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:57.985106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:24:57.985471 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:57.985449 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:24:57.985549 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:24:57.985517 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:01.985498696 +0000 UTC m=+44.526555116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:24:58.850540 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:58.850348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" event={"ID":"d7ae9a77-fee5-4386-9614-29fac591c309","Type":"ContainerStarted","Data":"68f617fdd4ac2cf947cd0f38f14e5bf7ece5d2581616b69c481debd08be8d323"} Jul 06 20:24:58.883461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:58.883418 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kxzmh" podStartSLOduration=8.05131563 podStartE2EDuration="39.883399404s" totalImagesPullingTime="31.832083774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:58.880966453 +0000 UTC m=+41.422022890" watchObservedRunningTime="2026-07-06 20:24:58.883399404 +0000 UTC m=+41.424455842" Jul 06 20:24:59.854497 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:59.854456 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h97f5" event={"ID":"6195ed3c-3037-4c3e-85c5-f381fce16615","Type":"ContainerStarted","Data":"16b9811115515148a53ed768036b00603924ff76dec7e2eb92dfd294f2790dee"} Jul 06 20:24:59.874832 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:24:59.874768 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h97f5" podStartSLOduration=2.945941142 podStartE2EDuration="5.87475293s" totalImagesPullingTime="2.928811788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:59.874583759 +0000 UTC m=+42.415640196" watchObservedRunningTime="2026-07-06 20:24:59.87475293 +0000 UTC m=+42.415809367" Jul 06 20:25:02.013973 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:02.013933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:25:02.014404 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:02.014065 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:25:02.014404 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:02.014116 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:10.014101827 +0000 UTC m=+52.555158242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:25:10.068225 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:10.068183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:25:10.068658 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:10.068313 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:25:10.068658 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:10.068388 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:26.068371229 +0000 UTC m=+68.609427655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:25:19.832179 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:19.831492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5qb8" Jul 06 20:25:24.255953 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.255912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:25:24.255953 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.255961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:25:24.258433 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.258412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:25:24.258549 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.258533 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:25:24.266595 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:24.266575 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 20:25:24.266675 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:24.266662 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs podName:1655b2df-cc8f-4633-87e9-1d5402e0ec7e nodeName:}" failed. No retries permitted until 2026-07-06 20:26:28.266638916 +0000 UTC m=+130.807695345 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs") pod "network-metrics-daemon-8jd6j" (UID: "1655b2df-cc8f-4633-87e9-1d5402e0ec7e") : secret "metrics-daemon-secret" not found Jul 06 20:25:24.270329 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.270310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f96f3b-f7be-4967-a67a-6b3952922001-original-pull-secret\") pod \"global-pull-secret-syncer-z2l27\" (UID: \"49f96f3b-f7be-4967-a67a-6b3952922001\") " pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:25:24.356612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.356569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:25:24.359297 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.359277 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:25:24.369703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.369685 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:25:24.380067 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.380043 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkb9\" (UniqueName: \"kubernetes.io/projected/f8d1a285-7254-46cb-8c67-83df6a73c9fd-kube-api-access-jfkb9\") pod \"network-check-target-f74sm\" (UID: \"f8d1a285-7254-46cb-8c67-83df6a73c9fd\") " pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:25:24.563112 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.563083 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2l27" Jul 06 20:25:24.669416 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.669388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lrkdh\"" Jul 06 20:25:24.677462 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.677433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:25:24.697502 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.697472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2l27"] Jul 06 20:25:24.700444 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:24.700405 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f96f3b_f7be_4967_a67a_6b3952922001.slice/crio-796d6973c02917daeab5407c43dcac255f3d452264355df7fa1cbf578dd4c037 WatchSource:0}: Error finding container 796d6973c02917daeab5407c43dcac255f3d452264355df7fa1cbf578dd4c037: Status 404 returned error can't find the container with id 796d6973c02917daeab5407c43dcac255f3d452264355df7fa1cbf578dd4c037 Jul 06 20:25:24.797455 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.797421 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f74sm"] Jul 06 20:25:24.800617 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:24.800593 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d1a285_7254_46cb_8c67_83df6a73c9fd.slice/crio-9337fa7f1fbe4af8193ad12303bcb5ab45ae0de6056e6df76ae27203c6b429f7 WatchSource:0}: Error finding container 9337fa7f1fbe4af8193ad12303bcb5ab45ae0de6056e6df76ae27203c6b429f7: Status 404 returned error can't find the container with id 9337fa7f1fbe4af8193ad12303bcb5ab45ae0de6056e6df76ae27203c6b429f7 Jul 06 20:25:24.913130 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.913027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f74sm" event={"ID":"f8d1a285-7254-46cb-8c67-83df6a73c9fd","Type":"ContainerStarted","Data":"9337fa7f1fbe4af8193ad12303bcb5ab45ae0de6056e6df76ae27203c6b429f7"} Jul 06 20:25:24.914104 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:24.914078 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2l27" event={"ID":"49f96f3b-f7be-4967-a67a-6b3952922001","Type":"ContainerStarted","Data":"796d6973c02917daeab5407c43dcac255f3d452264355df7fa1cbf578dd4c037"} Jul 06 20:25:26.070533 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:26.070489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:25:26.071135 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:26.070692 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:25:26.071135 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:26.070776 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:58.070752417 +0000 UTC m=+100.611808847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:25:29.928971 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:29.928933 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f74sm" event={"ID":"f8d1a285-7254-46cb-8c67-83df6a73c9fd","Type":"ContainerStarted","Data":"a82ac861344e94a6b70dbafe23630affa0fb82f6e945dfcf1b125edc27ab6e3e"} Jul 06 20:25:29.929468 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:29.929026 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:25:29.930537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:29.930507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2l27" event={"ID":"49f96f3b-f7be-4967-a67a-6b3952922001","Type":"ContainerStarted","Data":"43bbebe0819a5ad93400f464f9993f616add5d98fb889c7af05139c46708c4c2"} Jul 06 20:25:29.948734 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:29.948687 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f74sm" podStartSLOduration=66.52098754 podStartE2EDuration="1m10.948673461s" totalImagesPullingTime="4.427685921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:29.948338385 +0000 UTC m=+72.489394822" watchObservedRunningTime="2026-07-06 20:25:29.948673461 +0000 UTC m=+72.489729897" Jul 06 20:25:29.967112 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:29.967050 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z2l27" podStartSLOduration=66.436888214 podStartE2EDuration="1m10.967039281s" totalImagesPullingTime="4.530151067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:29.965896593 +0000 UTC m=+72.506953042" watchObservedRunningTime="2026-07-06 20:25:29.967039281 +0000 UTC m=+72.508095718" Jul 06 20:25:48.163117 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.162994 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l"] Jul 06 20:25:48.166335 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.166318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.169758 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.169735 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 20:25:48.169955 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.169743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sn246\"" Jul 06 20:25:48.171245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.171212 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:25:48.171354 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.171218 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:25:48.171486 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.171437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 20:25:48.175305 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.175283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l"] Jul 06 20:25:48.228490 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.228449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.228490 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.228488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.228715 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.228508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9gw\" (UniqueName: \"kubernetes.io/projected/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-kube-api-access-nh9gw\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.329629 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.329587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.329629 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.329628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.329849 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.329654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9gw\" (UniqueName: \"kubernetes.io/projected/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-kube-api-access-nh9gw\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.329849 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:48.329745 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:48.329849 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:48.329833 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:48.829817064 +0000 UTC m=+91.370873483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:48.330396 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.330376 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.346021 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.345988 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9gw\" (UniqueName: \"kubernetes.io/projected/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-kube-api-access-nh9gw\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.834011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:48.833971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:48.834230 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:48.834128 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:48.834279 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:48.834239 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:49.834221432 +0000 UTC m=+92.375277851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:49.842573 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:49.842534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:49.843012 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:49.842679 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:49.843012 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:49.842763 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:51.84274212 +0000 UTC m=+94.383798542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:51.856362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:51.856318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:51.856775 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:51.856449 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:51.856775 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:51.856503 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:55.856490436 +0000 UTC m=+98.397546857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:52.533726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.533693 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8lxbj"] Jul 06 20:25:52.535644 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.535629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.538228 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.538204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:25:52.538478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.538464 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:25:52.538527 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.538510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:25:52.538744 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.538726 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:25:52.539285 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.539267 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lmvxt\"" Jul 06 20:25:52.555897 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.555874 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8lxbj"] Jul 06 20:25:52.660504 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.660464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab0299c6-c4e2-4e1c-885c-a86f76517386-crio-socket\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.660504 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.660507 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0299c6-c4e2-4e1c-885c-a86f76517386-data-volume\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.660731 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.660534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxs2\" (UniqueName: \"kubernetes.io/projected/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-api-access-mtxs2\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.660731 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.660611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.660731 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.660648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.761972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.761940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab0299c6-c4e2-4e1c-885c-a86f76517386-crio-socket\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.761979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0299c6-c4e2-4e1c-885c-a86f76517386-data-volume\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.762003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxs2\" (UniqueName: \"kubernetes.io/projected/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-api-access-mtxs2\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.762038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.762057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.762073 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab0299c6-c4e2-4e1c-885c-a86f76517386-crio-socket\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.762300 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:52.762191 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:25:52.762300 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:52.762216 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:25:52.762300 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:52.762269 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:53.262252481 +0000 UTC m=+95.803308899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : configmap "kube-rbac-proxy" not found Jul 06 20:25:52.762300 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:52.762286 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:53.262277629 +0000 UTC m=+95.803334044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:25:52.762434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.762301 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab0299c6-c4e2-4e1c-885c-a86f76517386-data-volume\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:52.771254 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:52.771229 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxs2\" (UniqueName: \"kubernetes.io/projected/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-api-access-mtxs2\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:53.265416 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:53.265375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:53.265416 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:53.265419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:53.265921 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:53.265517 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:25:53.265921 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:53.265567 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:54.26555199 +0000 UTC m=+96.806608410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:25:53.265921 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:53.265512 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:25:53.265921 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:53.265675 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:54.265655064 +0000 UTC m=+96.806711496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : configmap "kube-rbac-proxy" not found Jul 06 20:25:54.226322 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.226288 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj"] Jul 06 20:25:54.228256 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.228241 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" Jul 06 20:25:54.230825 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.230800 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:25:54.230825 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.230807 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 20:25:54.231594 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.231577 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j8lr7\"" Jul 06 20:25:54.239205 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.239183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj"] Jul 06 20:25:54.272858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.272820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76ll\" (UniqueName: \"kubernetes.io/projected/20f76774-8494-48e8-a502-b6cdfc3d6c1d-kube-api-access-w76ll\") pod \"volume-data-source-validator-5d456fdbcd-85srj\" (UID: \"20f76774-8494-48e8-a502-b6cdfc3d6c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.272921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.272976 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.273010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.273042 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:56.273025711 +0000 UTC m=+98.814082126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : configmap "kube-rbac-proxy" not found Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.273097 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:25:54.273290 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.273141 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:56.273126938 +0000 UTC m=+98.814183352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:25:54.353490 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.353453 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk"] Jul 06 20:25:54.355431 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.355415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.358234 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.358215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 20:25:54.358376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.358354 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 20:25:54.358502 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.358487 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zfsh9\"" Jul 06 20:25:54.358572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.358488 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:25:54.372035 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.371997 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk"] Jul 06 20:25:54.374375 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.374351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76ll\" (UniqueName: \"kubernetes.io/projected/20f76774-8494-48e8-a502-b6cdfc3d6c1d-kube-api-access-w76ll\") pod \"volume-data-source-validator-5d456fdbcd-85srj\" (UID: \"20f76774-8494-48e8-a502-b6cdfc3d6c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" Jul 06 20:25:54.391240 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.391210 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76ll\" (UniqueName: \"kubernetes.io/projected/20f76774-8494-48e8-a502-b6cdfc3d6c1d-kube-api-access-w76ll\") pod \"volume-data-source-validator-5d456fdbcd-85srj\" (UID: \"20f76774-8494-48e8-a502-b6cdfc3d6c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" Jul 06 20:25:54.475475 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.475415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58k6x\" (UniqueName: \"kubernetes.io/projected/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-kube-api-access-58k6x\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.475655 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.475484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.519517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.519442 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8np8r_f7083172-62a8-4506-b77e-7112ccde6e98/dns-node-resolver/0.log" Jul 06 20:25:54.536552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.536512 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" Jul 06 20:25:54.576443 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.576408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.576641 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.576508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58k6x\" (UniqueName: \"kubernetes.io/projected/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-kube-api-access-58k6x\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.576641 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.576567 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:25:54.576760 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:54.576640 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:25:55.076622617 +0000 UTC m=+97.617679034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:25:54.589105 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.589079 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58k6x\" (UniqueName: \"kubernetes.io/projected/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-kube-api-access-58k6x\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:54.668460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.668431 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj"] Jul 06 20:25:54.671893 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:54.671868 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f76774_8494_48e8_a502_b6cdfc3d6c1d.slice/crio-88ec29251d49b6e086b2c08de1d12a54129747055bed153138579c17d604be75 WatchSource:0}: Error finding container 88ec29251d49b6e086b2c08de1d12a54129747055bed153138579c17d604be75: Status 404 returned error can't find the container with id 88ec29251d49b6e086b2c08de1d12a54129747055bed153138579c17d604be75 Jul 06 20:25:54.995668 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:54.995636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" event={"ID":"20f76774-8494-48e8-a502-b6cdfc3d6c1d","Type":"ContainerStarted","Data":"88ec29251d49b6e086b2c08de1d12a54129747055bed153138579c17d604be75"} Jul 06 20:25:55.080468 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.080433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:55.080586 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:55.080542 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:25:55.080623 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:55.080603 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:25:56.080589646 +0000 UTC m=+98.621646061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:25:55.226093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.226056 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hfd54"] Jul 06 20:25:55.227956 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.227941 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.232276 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.232254 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:25:55.232475 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.232461 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 20:25:55.232710 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.232694 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 20:25:55.232774 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.232698 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 20:25:55.232961 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.232945 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gkr8t\"" Jul 06 20:25:55.237941 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.237864 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 20:25:55.241741 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.241231 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hfd54"] Jul 06 20:25:55.282207 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.282088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-config\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.282207 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.282139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5krj2\" (UniqueName: \"kubernetes.io/projected/2d44e299-d22d-49d2-8ab4-b67020497397-kube-api-access-5krj2\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.282599 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.282268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d44e299-d22d-49d2-8ab4-b67020497397-serving-cert\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.282599 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.282299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-trusted-ca\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.313848 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.313821 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8rnd_212a155d-c62b-42a3-bbeb-d230b050d072/node-ca/0.log" Jul 06 20:25:55.383520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.383486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-config\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.383520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.383535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5krj2\" (UniqueName: \"kubernetes.io/projected/2d44e299-d22d-49d2-8ab4-b67020497397-kube-api-access-5krj2\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.383779 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.383586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d44e299-d22d-49d2-8ab4-b67020497397-serving-cert\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.383779 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.383602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-trusted-ca\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.384339 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.384312 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-config\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.384635 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.384612 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d44e299-d22d-49d2-8ab4-b67020497397-trusted-ca\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.386390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.386368 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d44e299-d22d-49d2-8ab4-b67020497397-serving-cert\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.394949 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.394920 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5krj2\" (UniqueName: \"kubernetes.io/projected/2d44e299-d22d-49d2-8ab4-b67020497397-kube-api-access-5krj2\") pod \"console-operator-68b757fb67-hfd54\" (UID: \"2d44e299-d22d-49d2-8ab4-b67020497397\") " pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.537788 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.537700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:25:55.817990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.817960 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-hfd54"] Jul 06 20:25:55.821117 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:55.821087 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d44e299_d22d_49d2_8ab4_b67020497397.slice/crio-4a8bd5ecc1d426cefbd1cba9daf86b12cb21aec65211803d78e125bda89fe57d WatchSource:0}: Error finding container 4a8bd5ecc1d426cefbd1cba9daf86b12cb21aec65211803d78e125bda89fe57d: Status 404 returned error can't find the container with id 4a8bd5ecc1d426cefbd1cba9daf86b12cb21aec65211803d78e125bda89fe57d Jul 06 20:25:55.888519 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.888479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:25:55.888698 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:55.888627 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:55.888698 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:55.888694 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:03.888678513 +0000 UTC m=+106.429734929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:25:55.913823 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.913780 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h97f5_6195ed3c-3037-4c3e-85c5-f381fce16615/serve-healthcheck-canary/0.log" Jul 06 20:25:55.999438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:55.999397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" event={"ID":"20f76774-8494-48e8-a502-b6cdfc3d6c1d","Type":"ContainerStarted","Data":"c67e2621052e468e7054db34d250e7ae94aa54c0b1c1371671e1d9d1761d519c"} Jul 06 20:25:56.000567 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:56.000545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" event={"ID":"2d44e299-d22d-49d2-8ab4-b67020497397","Type":"ContainerStarted","Data":"4a8bd5ecc1d426cefbd1cba9daf86b12cb21aec65211803d78e125bda89fe57d"} Jul 06 20:25:56.089800 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:56.089709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:56.089947 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.089845 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:25:56.089947 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.089899 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:25:58.089886009 +0000 UTC m=+100.630942424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:25:56.290728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:56.290691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:56.290728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:56.290727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:25:56.291191 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.290812 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:25:56.291191 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.290851 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:25:56.291191 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.290878 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:00.290862334 +0000 UTC m=+102.831918750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : configmap "kube-rbac-proxy" not found Jul 06 20:25:56.291191 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:56.290893 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:00.290887087 +0000 UTC m=+102.831943502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:25:58.105843 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.105801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:25:58.106418 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.105850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:25:58.106418 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:58.105959 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:25:58.106418 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:58.106014 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:25:58.106418 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:58.106026 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls podName:abb79635-08ac-4b40-93d2-f18435f585e8 nodeName:}" failed. No retries permitted until 2026-07-06 20:27:02.106007582 +0000 UTC m=+164.647064009 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls") pod "dns-default-4glv9" (UID: "abb79635-08ac-4b40-93d2-f18435f585e8") : secret "dns-default-metrics-tls" not found Jul 06 20:25:58.106418 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:25:58.106069 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:26:02.106054894 +0000 UTC m=+104.647111313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:25:58.216564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.216466 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-85srj" podStartSLOduration=3.181472163 podStartE2EDuration="4.216442903s" totalImagesPullingTime="1.03497074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:56.019172269 +0000 UTC m=+98.560228708" watchObservedRunningTime="2026-07-06 20:25:58.216442903 +0000 UTC m=+100.757499343" Jul 06 20:25:58.216720 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.216588 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx"] Jul 06 20:25:58.218594 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.218578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.221545 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.221518 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 20:25:58.222366 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.222341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8xwpk\"" Jul 06 20:25:58.222366 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.222358 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:25:58.222563 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.222339 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 20:25:58.222563 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.222341 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 20:25:58.239323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.239297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx"] Jul 06 20:25:58.307536 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.307498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/292b8f0b-f730-4fdd-a470-73e28ce6602f-config\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.307723 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.307563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/292b8f0b-f730-4fdd-a470-73e28ce6602f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.307723 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.307647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkp5\" (UniqueName: \"kubernetes.io/projected/292b8f0b-f730-4fdd-a470-73e28ce6602f-kube-api-access-vdkp5\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.315135 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.315108 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5"] Jul 06 20:25:58.317701 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.317688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.321381 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.321351 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:25:58.321381 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.321362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tb7vj\"" Jul 06 20:25:58.321797 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.321782 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 20:25:58.321865 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.321804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 20:25:58.321908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.321872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 20:25:58.333513 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.333492 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5"] Jul 06 20:25:58.408376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkp5\" (UniqueName: \"kubernetes.io/projected/292b8f0b-f730-4fdd-a470-73e28ce6602f-kube-api-access-vdkp5\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.408376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.408597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-kube-api-access-6br2k\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.408597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/292b8f0b-f730-4fdd-a470-73e28ce6602f-config\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.408597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.408597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.408578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/292b8f0b-f730-4fdd-a470-73e28ce6602f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.409044 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.409026 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/292b8f0b-f730-4fdd-a470-73e28ce6602f-config\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.410819 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.410800 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/292b8f0b-f730-4fdd-a470-73e28ce6602f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.418067 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.418045 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkp5\" (UniqueName: \"kubernetes.io/projected/292b8f0b-f730-4fdd-a470-73e28ce6602f-kube-api-access-vdkp5\") pod \"service-ca-operator-5d46d9ffd4-sbljx\" (UID: \"292b8f0b-f730-4fdd-a470-73e28ce6602f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.509376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.509282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.509376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.509324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-kube-api-access-6br2k\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.509376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.509346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.509872 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.509852 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.511608 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.511585 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.519787 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.519764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br2k\" (UniqueName: \"kubernetes.io/projected/fbb8cc97-039a-4f28-8c92-aaad7bf9e34f-kube-api-access-6br2k\") pod \"kube-storage-version-migrator-operator-846bf668f7-s9nl5\" (UID: \"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.527710 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.527686 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" Jul 06 20:25:58.626331 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.626299 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" Jul 06 20:25:58.658312 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.658279 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx"] Jul 06 20:25:58.661433 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:58.661379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b8f0b_f730_4fdd_a470_73e28ce6602f.slice/crio-3e2cd183ab7b7d2cb9f93c4d5fd38bca4c5cf0f2cd9360c6deb445b969349685 WatchSource:0}: Error finding container 3e2cd183ab7b7d2cb9f93c4d5fd38bca4c5cf0f2cd9360c6deb445b969349685: Status 404 returned error can't find the container with id 3e2cd183ab7b7d2cb9f93c4d5fd38bca4c5cf0f2cd9360c6deb445b969349685 Jul 06 20:25:58.749923 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:58.749746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5"] Jul 06 20:25:58.752351 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:25:58.752329 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb8cc97_039a_4f28_8c92_aaad7bf9e34f.slice/crio-622b54eb5d0c6b7ab18eacf89cbbaf7129402899b9a083ef5a6cc8456adbef60 WatchSource:0}: Error finding container 622b54eb5d0c6b7ab18eacf89cbbaf7129402899b9a083ef5a6cc8456adbef60: Status 404 returned error can't find the container with id 622b54eb5d0c6b7ab18eacf89cbbaf7129402899b9a083ef5a6cc8456adbef60 Jul 06 20:25:59.010901 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.010864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" event={"ID":"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f","Type":"ContainerStarted","Data":"622b54eb5d0c6b7ab18eacf89cbbaf7129402899b9a083ef5a6cc8456adbef60"} Jul 06 20:25:59.012419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.012393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/0.log" Jul 06 20:25:59.012508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.012443 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d44e299-d22d-49d2-8ab4-b67020497397" containerID="887adcb7e30167a69f64901fdeb0f4e769be474ff08109e79cdf2e0a5a9a3cd1" exitCode=255 Jul 06 20:25:59.012508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.012476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" event={"ID":"2d44e299-d22d-49d2-8ab4-b67020497397","Type":"ContainerDied","Data":"887adcb7e30167a69f64901fdeb0f4e769be474ff08109e79cdf2e0a5a9a3cd1"} Jul 06 20:25:59.012748 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.012727 2562 scope.go:117] "RemoveContainer" containerID="887adcb7e30167a69f64901fdeb0f4e769be474ff08109e79cdf2e0a5a9a3cd1" Jul 06 20:25:59.013741 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:25:59.013719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" event={"ID":"292b8f0b-f730-4fdd-a470-73e28ce6602f","Type":"ContainerStarted","Data":"3e2cd183ab7b7d2cb9f93c4d5fd38bca4c5cf0f2cd9360c6deb445b969349685"} Jul 06 20:26:00.019116 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019082 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/1.log" Jul 06 20:26:00.019553 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019536 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/0.log" Jul 06 20:26:00.019609 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019575 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d44e299-d22d-49d2-8ab4-b67020497397" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" exitCode=255 Jul 06 20:26:00.019667 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" event={"ID":"2d44e299-d22d-49d2-8ab4-b67020497397","Type":"ContainerDied","Data":"3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294"} Jul 06 20:26:00.019724 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019712 2562 scope.go:117] "RemoveContainer" containerID="887adcb7e30167a69f64901fdeb0f4e769be474ff08109e79cdf2e0a5a9a3cd1" Jul 06 20:26:00.019953 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.019931 2562 scope.go:117] "RemoveContainer" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" Jul 06 20:26:00.020164 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:00.020127 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:00.323682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.323648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:00.323859 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.323690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:00.323859 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:00.323782 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:26:00.323859 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:00.323852 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:08.323836616 +0000 UTC m=+110.864893034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : configmap "kube-rbac-proxy" not found Jul 06 20:26:00.323988 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:00.323873 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:26:00.323988 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:00.323934 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:08.323917201 +0000 UTC m=+110.864973623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:26:00.935306 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:00.935269 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f74sm" Jul 06 20:26:01.024196 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:01.024125 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/1.log" Jul 06 20:26:01.024668 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:01.024608 2562 scope.go:117] "RemoveContainer" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" Jul 06 20:26:01.024815 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:01.024793 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:02.020056 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.019968 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq"] Jul 06 20:26:02.040461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.040417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq"] Jul 06 20:26:02.040862 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.040464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" event={"ID":"292b8f0b-f730-4fdd-a470-73e28ce6602f","Type":"ContainerStarted","Data":"b68bc53efce50f9e01cdb6d25677dd67c08eb77bd01321c9a15367624aeb5a83"} Jul 06 20:26:02.040862 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.040489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" event={"ID":"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f","Type":"ContainerStarted","Data":"75277b0e4c0467098da0f8c68ca9f754730e2f63b8c828866e1ad98f73c6f391"} Jul 06 20:26:02.040862 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.040578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" Jul 06 20:26:02.042962 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.042936 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s8vwt\"" Jul 06 20:26:02.046843 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.046787 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" podStartSLOduration=1.009437093 podStartE2EDuration="4.046769293s" totalImagesPullingTime="3.0373322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:02.046330895 +0000 UTC m=+104.587387332" watchObservedRunningTime="2026-07-06 20:26:02.046769293 +0000 UTC m=+104.587825732" Jul 06 20:26:02.064018 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.063961 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" podStartSLOduration=1.110433754 podStartE2EDuration="4.063944318s" totalImagesPullingTime="2.953510564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:02.063732249 +0000 UTC m=+104.604788686" watchObservedRunningTime="2026-07-06 20:26:02.063944318 +0000 UTC m=+104.605000792" Jul 06 20:26:02.140123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.140077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:26:02.140646 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.140619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqtc\" (UniqueName: \"kubernetes.io/projected/88603659-b277-4797-adec-51a4b9eb65d4-kube-api-access-whqtc\") pod \"network-check-source-989f8b8dc-t7llq\" (UID: \"88603659-b277-4797-adec-51a4b9eb65d4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" Jul 06 20:26:02.140829 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:02.140812 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:26:02.140915 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:02.140904 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:26:10.140884397 +0000 UTC m=+112.681940821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:26:02.241922 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.241877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqtc\" (UniqueName: \"kubernetes.io/projected/88603659-b277-4797-adec-51a4b9eb65d4-kube-api-access-whqtc\") pod \"network-check-source-989f8b8dc-t7llq\" (UID: \"88603659-b277-4797-adec-51a4b9eb65d4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" Jul 06 20:26:02.250961 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.250924 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqtc\" (UniqueName: \"kubernetes.io/projected/88603659-b277-4797-adec-51a4b9eb65d4-kube-api-access-whqtc\") pod \"network-check-source-989f8b8dc-t7llq\" (UID: \"88603659-b277-4797-adec-51a4b9eb65d4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" Jul 06 20:26:02.351029 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.350990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" Jul 06 20:26:02.502309 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:02.501903 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq"] Jul 06 20:26:02.504826 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:02.504795 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88603659_b277_4797_adec_51a4b9eb65d4.slice/crio-fa2673745e2fb842c531d45187b90c328b02bef0c5e6f53d4b7d334627921188 WatchSource:0}: Error finding container fa2673745e2fb842c531d45187b90c328b02bef0c5e6f53d4b7d334627921188: Status 404 returned error can't find the container with id fa2673745e2fb842c531d45187b90c328b02bef0c5e6f53d4b7d334627921188 Jul 06 20:26:03.036721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:03.036680 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" event={"ID":"88603659-b277-4797-adec-51a4b9eb65d4","Type":"ContainerStarted","Data":"7190d25c1cb13ebe9dbca4e68b94e478dbf860a3e670e4adf65f4e24d2ca9f6c"} Jul 06 20:26:03.036929 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:03.036745 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" event={"ID":"88603659-b277-4797-adec-51a4b9eb65d4","Type":"ContainerStarted","Data":"fa2673745e2fb842c531d45187b90c328b02bef0c5e6f53d4b7d334627921188"} Jul 06 20:26:03.054705 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:03.054644 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-t7llq" podStartSLOduration=2.054622325 podStartE2EDuration="2.054622325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:03.05432309 +0000 UTC m=+105.595379522" watchObservedRunningTime="2026-07-06 20:26:03.054622325 +0000 UTC m=+105.595678765" Jul 06 20:26:03.955435 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:03.955403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:26:03.955652 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:03.955576 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:26:03.955716 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:03.955658 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls podName:f9c8d0b5-f397-4581-b6bc-16947be0e6c1 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:19.955637462 +0000 UTC m=+122.496693891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6sc6l" (UID: "f9c8d0b5-f397-4581-b6bc-16947be0e6c1") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:26:05.538324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:05.538260 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:05.538324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:05.538340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:05.538768 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:05.538694 2562 scope.go:117] "RemoveContainer" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" Jul 06 20:26:05.538856 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:05.538835 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:08.391007 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:08.390959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:08.391007 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:08.391009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:08.391482 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:08.391106 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:26:08.391482 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:08.391183 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls podName:ab0299c6-c4e2-4e1c-885c-a86f76517386 nodeName:}" failed. No retries permitted until 2026-07-06 20:26:24.39114402 +0000 UTC m=+126.932200437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8lxbj" (UID: "ab0299c6-c4e2-4e1c-885c-a86f76517386") : secret "insights-runtime-extractor-tls" not found Jul 06 20:26:08.391561 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:08.391512 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab0299c6-c4e2-4e1c-885c-a86f76517386-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:10.205895 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:10.205842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:26:10.206322 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:10.206003 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 20:26:10.206322 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:10.206080 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls podName:77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a nodeName:}" failed. No retries permitted until 2026-07-06 20:26:26.206063868 +0000 UTC m=+128.747120283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5xpk" (UID: "77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a") : secret "samples-operator-tls" not found Jul 06 20:26:19.552854 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:19.552820 2562 scope.go:117] "RemoveContainer" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" Jul 06 20:26:19.984526 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:19.984483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:26:19.986843 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:19.986813 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9c8d0b5-f397-4581-b6bc-16947be0e6c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6sc6l\" (UID: \"f9c8d0b5-f397-4581-b6bc-16947be0e6c1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:26:20.090886 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.090851 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:26:20.091259 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.091243 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/1.log" Jul 06 20:26:20.091312 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.091279 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d44e299-d22d-49d2-8ab4-b67020497397" containerID="8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d" exitCode=255 Jul 06 20:26:20.091362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.091321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" event={"ID":"2d44e299-d22d-49d2-8ab4-b67020497397","Type":"ContainerDied","Data":"8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d"} Jul 06 20:26:20.091394 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.091373 2562 scope.go:117] "RemoveContainer" containerID="3adfb8899381c5f721e974fc323d4e2d5169877f3e650f6d2290530da6073294" Jul 06 20:26:20.091720 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.091692 2562 scope.go:117] "RemoveContainer" containerID="8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d" Jul 06 20:26:20.091881 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:20.091860 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:20.277776 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.277684 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sn246\"" Jul 06 20:26:20.285587 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.285567 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" Jul 06 20:26:20.409420 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:20.409395 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l"] Jul 06 20:26:20.411629 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:20.411599 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c8d0b5_f397_4581_b6bc_16947be0e6c1.slice/crio-ce69e4ef6fb5cb5d4bbb7d0f68013ccd1045575d8d2c9820d869db0d3e3d6bde WatchSource:0}: Error finding container ce69e4ef6fb5cb5d4bbb7d0f68013ccd1045575d8d2c9820d869db0d3e3d6bde: Status 404 returned error can't find the container with id ce69e4ef6fb5cb5d4bbb7d0f68013ccd1045575d8d2c9820d869db0d3e3d6bde Jul 06 20:26:21.096985 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:21.096952 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:26:21.098094 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:21.098063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" event={"ID":"f9c8d0b5-f397-4581-b6bc-16947be0e6c1","Type":"ContainerStarted","Data":"ce69e4ef6fb5cb5d4bbb7d0f68013ccd1045575d8d2c9820d869db0d3e3d6bde"} Jul 06 20:26:23.105511 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:23.105472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" event={"ID":"f9c8d0b5-f397-4581-b6bc-16947be0e6c1","Type":"ContainerStarted","Data":"c83d5fc105b57e1bb90e37ab410c85864d2a33c40d7c68f0b1ff23324b9863f3"} Jul 06 20:26:23.126223 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:23.126170 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6sc6l" podStartSLOduration=33.169157167 podStartE2EDuration="35.126140207s" totalImagesPullingTime="1.95698304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:23.124762189 +0000 UTC m=+125.665818623" watchObservedRunningTime="2026-07-06 20:26:23.126140207 +0000 UTC m=+125.667196644" Jul 06 20:26:24.420341 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:24.420301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:24.422584 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:24.422548 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab0299c6-c4e2-4e1c-885c-a86f76517386-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8lxbj\" (UID: \"ab0299c6-c4e2-4e1c-885c-a86f76517386\") " pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:24.647499 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:24.647465 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lmvxt\"" Jul 06 20:26:24.655660 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:24.655636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8lxbj" Jul 06 20:26:24.801137 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:24.801103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8lxbj"] Jul 06 20:26:24.804498 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:24.804459 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0299c6_c4e2_4e1c_885c_a86f76517386.slice/crio-9f45be4cafd3468d44bdec37566203258156b74d8b9950a17541c2b455df35d9 WatchSource:0}: Error finding container 9f45be4cafd3468d44bdec37566203258156b74d8b9950a17541c2b455df35d9: Status 404 returned error can't find the container with id 9f45be4cafd3468d44bdec37566203258156b74d8b9950a17541c2b455df35d9 Jul 06 20:26:25.113372 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:25.113335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lxbj" event={"ID":"ab0299c6-c4e2-4e1c-885c-a86f76517386","Type":"ContainerStarted","Data":"9f45be4cafd3468d44bdec37566203258156b74d8b9950a17541c2b455df35d9"} Jul 06 20:26:25.538599 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:25.538429 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:25.538599 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:25.538512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:25.539111 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:25.538909 2562 scope.go:117] "RemoveContainer" containerID="8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d" Jul 06 20:26:25.539111 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:25.539090 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:26.244429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:26.244393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:26:26.246984 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:26.246955 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5xpk\" (UID: \"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:26:26.466168 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:26.466111 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zfsh9\"" Jul 06 20:26:26.474522 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:26.474266 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" Jul 06 20:26:27.717552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.717506 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb"] Jul 06 20:26:27.726482 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.726456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.729073 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.729049 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 20:26:27.729793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.729771 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q2lp9\"" Jul 06 20:26:27.729903 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.729792 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 20:26:27.739388 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.739363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb"] Jul 06 20:26:27.755714 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.755682 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dcb484c46-mmn49"] Jul 06 20:26:27.756011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.755982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a51e4603-2179-4622-bed7-8cfdd1a920fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.756171 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.756043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a51e4603-2179-4622-bed7-8cfdd1a920fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.760547 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.760526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.763387 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.763347 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mdtgg\"" Jul 06 20:26:27.763573 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.763550 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:26:27.763910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.763891 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:26:27.764011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.763914 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:26:27.769826 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.769790 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:26:27.778517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.778469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcb484c46-mmn49"] Jul 06 20:26:27.799587 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.799552 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7"] Jul 06 20:26:27.805954 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.805604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:27.809361 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.809332 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 20:26:27.809884 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.809818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-492dz\"" Jul 06 20:26:27.821596 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.821078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7"] Jul 06 20:26:27.856589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-certificates\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856780 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss595\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-kube-api-access-ss595\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856780 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e0099c-0f37-42d6-9ac3-181ed259799c-ca-trust-extracted\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856780 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-trusted-ca\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856780 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856771 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a51e4603-2179-4622-bed7-8cfdd1a920fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.856987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480046e-7b9f-4a9f-918b-a32b302265c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-296c7\" (UID: \"b480046e-7b9f-4a9f-918b-a32b302265c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:27.856987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-installation-pull-secrets\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-image-registry-private-configuration\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.856987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a51e4603-2179-4622-bed7-8cfdd1a920fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.856987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.856970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-tls\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.857241 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.857021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-bound-sa-token\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.859204 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.859119 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a51e4603-2179-4622-bed7-8cfdd1a920fa-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.864440 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.862070 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a51e4603-2179-4622-bed7-8cfdd1a920fa-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-tkfrb\" (UID: \"a51e4603-2179-4622-bed7-8cfdd1a920fa\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:27.958429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-trusted-ca\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480046e-7b9f-4a9f-918b-a32b302265c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-296c7\" (UID: \"b480046e-7b9f-4a9f-918b-a32b302265c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-installation-pull-secrets\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-image-registry-private-configuration\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-tls\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-bound-sa-token\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-certificates\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958960 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss595\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-kube-api-access-ss595\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.958960 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.958756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e0099c-0f37-42d6-9ac3-181ed259799c-ca-trust-extracted\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.959487 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.959426 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58e0099c-0f37-42d6-9ac3-181ed259799c-ca-trust-extracted\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.959581 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.959489 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-trusted-ca\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.959698 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.959646 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-certificates\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.961605 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.961581 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-image-registry-private-configuration\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.961784 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.961757 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58e0099c-0f37-42d6-9ac3-181ed259799c-installation-pull-secrets\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.961880 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.961803 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480046e-7b9f-4a9f-918b-a32b302265c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-296c7\" (UID: \"b480046e-7b9f-4a9f-918b-a32b302265c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:27.962310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.962285 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-registry-tls\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.967912 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.967852 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-bound-sa-token\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:27.968424 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:27.968404 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss595\" (UniqueName: \"kubernetes.io/projected/58e0099c-0f37-42d6-9ac3-181ed259799c-kube-api-access-ss595\") pod \"image-registry-6dcb484c46-mmn49\" (UID: \"58e0099c-0f37-42d6-9ac3-181ed259799c\") " pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:28.036826 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.036786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" Jul 06 20:26:28.071624 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.071590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:28.121581 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.121549 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:28.362576 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.362539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:26:28.365195 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.365162 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1655b2df-cc8f-4633-87e9-1d5402e0ec7e-metrics-certs\") pod \"network-metrics-daemon-8jd6j\" (UID: \"1655b2df-cc8f-4633-87e9-1d5402e0ec7e\") " pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:26:28.565023 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.564840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsjzg\"" Jul 06 20:26:28.572795 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.572769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jd6j" Jul 06 20:26:28.627356 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.627202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcb484c46-mmn49"] Jul 06 20:26:28.632505 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:28.632435 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e0099c_0f37_42d6_9ac3_181ed259799c.slice/crio-b4f3956addc0686e6965faea87e8b820c7451fcd797f35c7ef79ff504deba590 WatchSource:0}: Error finding container b4f3956addc0686e6965faea87e8b820c7451fcd797f35c7ef79ff504deba590: Status 404 returned error can't find the container with id b4f3956addc0686e6965faea87e8b820c7451fcd797f35c7ef79ff504deba590 Jul 06 20:26:28.642690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.642629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk"] Jul 06 20:26:28.743860 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.743746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jd6j"] Jul 06 20:26:28.746330 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:28.746300 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1655b2df_cc8f_4633_87e9_1d5402e0ec7e.slice/crio-8c8759beb0b43fdd9364209079c5f43ebd7cc9ca2807508a22eb9ca60e1f3613 WatchSource:0}: Error finding container 8c8759beb0b43fdd9364209079c5f43ebd7cc9ca2807508a22eb9ca60e1f3613: Status 404 returned error can't find the container with id 8c8759beb0b43fdd9364209079c5f43ebd7cc9ca2807508a22eb9ca60e1f3613 Jul 06 20:26:28.865896 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.865690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7"] Jul 06 20:26:28.867595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:28.867565 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb"] Jul 06 20:26:28.871849 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:28.871814 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb480046e_7b9f_4a9f_918b_a32b302265c7.slice/crio-5e95dd023780be5e40ee34e2eae9ac638be28d5d6dd6587c626a14f4ba21d49c WatchSource:0}: Error finding container 5e95dd023780be5e40ee34e2eae9ac638be28d5d6dd6587c626a14f4ba21d49c: Status 404 returned error can't find the container with id 5e95dd023780be5e40ee34e2eae9ac638be28d5d6dd6587c626a14f4ba21d49c Jul 06 20:26:28.872527 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:28.872465 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51e4603_2179_4622_bed7_8cfdd1a920fa.slice/crio-a2b6f633f88bb164e41a95b6c2254a06c3050390f78b9a468ebc1e1dd3f178eb WatchSource:0}: Error finding container a2b6f633f88bb164e41a95b6c2254a06c3050390f78b9a468ebc1e1dd3f178eb: Status 404 returned error can't find the container with id a2b6f633f88bb164e41a95b6c2254a06c3050390f78b9a468ebc1e1dd3f178eb Jul 06 20:26:29.129178 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.129068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" event={"ID":"a51e4603-2179-4622-bed7-8cfdd1a920fa","Type":"ContainerStarted","Data":"a2b6f633f88bb164e41a95b6c2254a06c3050390f78b9a468ebc1e1dd3f178eb"} Jul 06 20:26:29.130849 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.130806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lxbj" event={"ID":"ab0299c6-c4e2-4e1c-885c-a86f76517386","Type":"ContainerStarted","Data":"28c5ab70ea1d3b464d41de157ed2abb35d9b0596aab71708091fb103371e0588"} Jul 06 20:26:29.132326 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.132296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jd6j" event={"ID":"1655b2df-cc8f-4633-87e9-1d5402e0ec7e","Type":"ContainerStarted","Data":"8c8759beb0b43fdd9364209079c5f43ebd7cc9ca2807508a22eb9ca60e1f3613"} Jul 06 20:26:29.134197 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.134168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" event={"ID":"58e0099c-0f37-42d6-9ac3-181ed259799c","Type":"ContainerStarted","Data":"c73299d43a505605b7b2c40976bcd3cfc3d36352a541bb985ea71ec74aa5be7c"} Jul 06 20:26:29.134311 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.134221 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" event={"ID":"58e0099c-0f37-42d6-9ac3-181ed259799c","Type":"ContainerStarted","Data":"b4f3956addc0686e6965faea87e8b820c7451fcd797f35c7ef79ff504deba590"} Jul 06 20:26:29.134311 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.134261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:26:29.135735 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.135706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" event={"ID":"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a","Type":"ContainerStarted","Data":"30a786258b6a93b212df984dafa0ac4837e8ab6cd54a0fac7a7e287e21411948"} Jul 06 20:26:29.137000 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.136975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" event={"ID":"b480046e-7b9f-4a9f-918b-a32b302265c7","Type":"ContainerStarted","Data":"5e95dd023780be5e40ee34e2eae9ac638be28d5d6dd6587c626a14f4ba21d49c"} Jul 06 20:26:29.164715 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:29.164659 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" podStartSLOduration=2.164638372 podStartE2EDuration="2.164638372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:29.164440226 +0000 UTC m=+131.705496666" watchObservedRunningTime="2026-07-06 20:26:29.164638372 +0000 UTC m=+131.705694810" Jul 06 20:26:30.146098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:30.146047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lxbj" event={"ID":"ab0299c6-c4e2-4e1c-885c-a86f76517386","Type":"ContainerStarted","Data":"f6cb81fa9847b0c03d285a8b216fa0b7599d95845df22cc01d0632bcc2b8b7dc"} Jul 06 20:26:31.155283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.154311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" event={"ID":"b480046e-7b9f-4a9f-918b-a32b302265c7","Type":"ContainerStarted","Data":"ec2737b3dbff2e80019c8aabc40d164d394acdb3d71bfd11990058c0b8af1df7"} Jul 06 20:26:31.155283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.155182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:31.157656 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.157524 2562 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-d559d66d5-296c7 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/healthz\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body= Jul 06 20:26:31.157656 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.157577 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" podUID="b480046e-7b9f-4a9f-918b-a32b302265c7" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.18:8443/healthz\": dial tcp 10.133.0.18:8443: connect: connection refused" Jul 06 20:26:31.159432 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.159406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" event={"ID":"a51e4603-2179-4622-bed7-8cfdd1a920fa","Type":"ContainerStarted","Data":"7aeade72dd31a41bbf7892fddfc2021df10a425009f55fa9f7ece561ba07c76f"} Jul 06 20:26:31.162805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.162441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jd6j" event={"ID":"1655b2df-cc8f-4633-87e9-1d5402e0ec7e","Type":"ContainerStarted","Data":"f9b3da595ce2b52969d8b1b9bf0259af21915ae19504706fc8a7943f1ca10f3f"} Jul 06 20:26:31.164395 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.164328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" event={"ID":"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a","Type":"ContainerStarted","Data":"47835d8dd9170d289f8928201f3a9f3c198b5a14a72e56ce9fbe0e980870e3ac"} Jul 06 20:26:31.193792 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.193732 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-tkfrb" podStartSLOduration=2.050502947 podStartE2EDuration="4.193711755s" totalImagesPullingTime="2.143208808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:31.192586538 +0000 UTC m=+133.733642978" watchObservedRunningTime="2026-07-06 20:26:31.193711755 +0000 UTC m=+133.734768193" Jul 06 20:26:31.194512 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:31.194458 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" podStartSLOduration=2.050141759 podStartE2EDuration="4.19444286s" totalImagesPullingTime="2.144301101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:31.174528553 +0000 UTC m=+133.715584993" watchObservedRunningTime="2026-07-06 20:26:31.19444286 +0000 UTC m=+133.735499298" Jul 06 20:26:32.170589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.170545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" event={"ID":"77bf517b-bfbf-4f0f-abf9-a5b7ec786e8a","Type":"ContainerStarted","Data":"2189b009b2cf403ffdd0da1ad2657f0576f3d8af9b3e8b0aabc09ce39bea5085"} Jul 06 20:26:32.172450 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.172361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jd6j" event={"ID":"1655b2df-cc8f-4633-87e9-1d5402e0ec7e","Type":"ContainerStarted","Data":"182c99f9c2725d164222d381d8359a9c5322f23bf4f211f9ce4d5bf94db59832"} Jul 06 20:26:32.177288 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.177264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-296c7" Jul 06 20:26:32.189537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.189491 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5xpk" podStartSLOduration=35.858491904 podStartE2EDuration="38.189475848s" totalImagesPullingTime="2.330983944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:32.188077724 +0000 UTC m=+134.729134161" watchObservedRunningTime="2026-07-06 20:26:32.189475848 +0000 UTC m=+134.730532285" Jul 06 20:26:32.211900 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.211835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8jd6j" podStartSLOduration=130.941015593 podStartE2EDuration="2m13.211811967s" totalImagesPullingTime="2.270796374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:32.210731662 +0000 UTC m=+134.751788111" watchObservedRunningTime="2026-07-06 20:26:32.211811967 +0000 UTC m=+134.752868404" Jul 06 20:26:32.976715 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.976677 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm"] Jul 06 20:26:32.979876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.979860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:32.982506 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.982485 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 20:26:32.982618 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.982486 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 20:26:32.983169 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.983134 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:26:32.983301 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.983286 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-22x7x\"" Jul 06 20:26:32.990688 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:32.990667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm"] Jul 06 20:26:33.004676 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.004652 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtbp\" (UniqueName: \"kubernetes.io/projected/afc47ba1-4245-4770-a5d5-562da03326aa-kube-api-access-6gtbp\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.004818 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.004685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc47ba1-4245-4770-a5d5-562da03326aa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.004818 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.004784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.004917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.004833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.106391 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.105501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.106572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.106443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.106572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.106508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtbp\" (UniqueName: \"kubernetes.io/projected/afc47ba1-4245-4770-a5d5-562da03326aa-kube-api-access-6gtbp\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.106572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.106548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc47ba1-4245-4770-a5d5-562da03326aa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.106718 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:33.106655 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 20:26:33.106765 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:33.106733 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls podName:afc47ba1-4245-4770-a5d5-562da03326aa nodeName:}" failed. No retries permitted until 2026-07-06 20:26:33.606713533 +0000 UTC m=+136.147769961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-7x4mm" (UID: "afc47ba1-4245-4770-a5d5-562da03326aa") : secret "prometheus-operator-tls" not found Jul 06 20:26:33.109857 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.107342 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc47ba1-4245-4770-a5d5-562da03326aa-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.110688 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.110666 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.124827 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.124794 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtbp\" (UniqueName: \"kubernetes.io/projected/afc47ba1-4245-4770-a5d5-562da03326aa-kube-api-access-6gtbp\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.177401 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.177359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8lxbj" event={"ID":"ab0299c6-c4e2-4e1c-885c-a86f76517386","Type":"ContainerStarted","Data":"7cb9d872142d73706145000ad68019a0eab051e997f320bb0d321e2f2adc0a91"} Jul 06 20:26:33.200323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.200276 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8lxbj" podStartSLOduration=33.440742365 podStartE2EDuration="41.200259415s" totalImagesPullingTime="7.75951705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:25:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:33.198952479 +0000 UTC m=+135.740008928" watchObservedRunningTime="2026-07-06 20:26:33.200259415 +0000 UTC m=+135.741315882" Jul 06 20:26:33.612708 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.612670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.615014 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.614994 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/afc47ba1-4245-4770-a5d5-562da03326aa-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7x4mm\" (UID: \"afc47ba1-4245-4770-a5d5-562da03326aa\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:33.889011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:33.888909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" Jul 06 20:26:34.015441 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:34.015326 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm"] Jul 06 20:26:34.017867 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:34.017843 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc47ba1_4245_4770_a5d5_562da03326aa.slice/crio-89a4b35b8bf5378b2479e0ad1b5b1454848771c9e4161ce0e1cb95d5193ff996 WatchSource:0}: Error finding container 89a4b35b8bf5378b2479e0ad1b5b1454848771c9e4161ce0e1cb95d5193ff996: Status 404 returned error can't find the container with id 89a4b35b8bf5378b2479e0ad1b5b1454848771c9e4161ce0e1cb95d5193ff996 Jul 06 20:26:34.182065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:34.181974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" event={"ID":"afc47ba1-4245-4770-a5d5-562da03326aa","Type":"ContainerStarted","Data":"89a4b35b8bf5378b2479e0ad1b5b1454848771c9e4161ce0e1cb95d5193ff996"} Jul 06 20:26:36.191370 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:36.191331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" event={"ID":"afc47ba1-4245-4770-a5d5-562da03326aa","Type":"ContainerStarted","Data":"3fed30880ef982f6646e55b1bb8efd5d54f604464f1637f06dd705fbb47b6f01"} Jul 06 20:26:36.191762 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:36.191394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" event={"ID":"afc47ba1-4245-4770-a5d5-562da03326aa","Type":"ContainerStarted","Data":"77ab1c552628c05ad544f6dff83bac13f32058711f4787fa8534aac823e4cc74"} Jul 06 20:26:36.210169 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:36.210103 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7x4mm" podStartSLOduration=2.854860516 podStartE2EDuration="4.210087393s" totalImagesPullingTime="1.355226877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:36.208590806 +0000 UTC m=+138.749647236" watchObservedRunningTime="2026-07-06 20:26:36.210087393 +0000 UTC m=+138.751143863" Jul 06 20:26:38.341591 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.341560 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n"] Jul 06 20:26:38.344718 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.344700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.348267 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.348244 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 20:26:38.352204 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.352139 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 20:26:38.352670 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.352647 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rlc45\"" Jul 06 20:26:38.357926 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.357896 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n"] Jul 06 20:26:38.381462 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.381433 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5l2ht"] Jul 06 20:26:38.384595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.384575 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.387236 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.387212 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:26:38.387463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.387441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:26:38.387523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.387463 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-glmh5\"" Jul 06 20:26:38.387615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.387599 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:26:38.454539 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454698 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454698 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e67839-19c2-47fc-92d6-10f8a7a517cc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.454698 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-wtmp\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454698 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.454841 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgd7\" (UniqueName: \"kubernetes.io/projected/06e67839-19c2-47fc-92d6-10f8a7a517cc-kube-api-access-nmgd7\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.454841 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xftq\" (UniqueName: \"kubernetes.io/projected/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-kube-api-access-9xftq\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454841 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.454931 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-textfile\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454931 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-metrics-client-ca\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454931 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-root\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.454931 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-sys\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.455042 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.454944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.555571 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.555571 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.555808 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:38.555692 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 20:26:38.555808 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e67839-19c2-47fc-92d6-10f8a7a517cc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.555808 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:38.555763 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls podName:d31fefe3-4d3c-4109-aa75-7ca9818d6e4b nodeName:}" failed. No retries permitted until 2026-07-06 20:26:39.05574153 +0000 UTC m=+141.596797945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls") pod "node-exporter-5l2ht" (UID: "d31fefe3-4d3c-4109-aa75-7ca9818d6e4b") : secret "node-exporter-tls" not found Jul 06 20:26:38.555808 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-wtmp\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.556017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgd7\" (UniqueName: \"kubernetes.io/projected/06e67839-19c2-47fc-92d6-10f8a7a517cc-kube-api-access-nmgd7\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.556017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xftq\" (UniqueName: \"kubernetes.io/projected/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-kube-api-access-9xftq\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555980 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-wtmp\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.555993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.556286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-textfile\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-metrics-client-ca\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-root\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-sys\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556511 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556320 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-root\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556511 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556323 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-sys\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556644 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556621 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-textfile\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.556704 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e67839-19c2-47fc-92d6-10f8a7a517cc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.556990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.556954 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.557213 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.557192 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-metrics-client-ca\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.558348 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.558321 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.558564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.558540 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.559065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.559045 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06e67839-19c2-47fc-92d6-10f8a7a517cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.564578 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.564556 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xftq\" (UniqueName: \"kubernetes.io/projected/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-kube-api-access-9xftq\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:38.564672 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.564616 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgd7\" (UniqueName: \"kubernetes.io/projected/06e67839-19c2-47fc-92d6-10f8a7a517cc-kube-api-access-nmgd7\") pod \"openshift-state-metrics-54cf96dfcf-cnn6n\" (UID: \"06e67839-19c2-47fc-92d6-10f8a7a517cc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.658002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.657925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" Jul 06 20:26:38.791499 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:38.791469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n"] Jul 06 20:26:38.794719 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:38.794692 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e67839_19c2_47fc_92d6_10f8a7a517cc.slice/crio-e5fd1664539ca2915c82f6808dbc9cd157ead3abce5a13a9c2090e21fd4da8ee WatchSource:0}: Error finding container e5fd1664539ca2915c82f6808dbc9cd157ead3abce5a13a9c2090e21fd4da8ee: Status 404 returned error can't find the container with id e5fd1664539ca2915c82f6808dbc9cd157ead3abce5a13a9c2090e21fd4da8ee Jul 06 20:26:39.060081 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.060046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:39.062358 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.062334 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d31fefe3-4d3c-4109-aa75-7ca9818d6e4b-node-exporter-tls\") pod \"node-exporter-5l2ht\" (UID: \"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b\") " pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:39.203878 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.203795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" event={"ID":"06e67839-19c2-47fc-92d6-10f8a7a517cc","Type":"ContainerStarted","Data":"1cb34390851db2c57b7ff9b51d63cea7cd7b324a4649cb941a5f0161e9883cfd"} Jul 06 20:26:39.203878 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.203852 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" event={"ID":"06e67839-19c2-47fc-92d6-10f8a7a517cc","Type":"ContainerStarted","Data":"80184e2ccff83e0704561de24184551896b9df9668d0b9d16cdf3fa6d2a2ee1a"} Jul 06 20:26:39.203878 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.203871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" event={"ID":"06e67839-19c2-47fc-92d6-10f8a7a517cc","Type":"ContainerStarted","Data":"e5fd1664539ca2915c82f6808dbc9cd157ead3abce5a13a9c2090e21fd4da8ee"} Jul 06 20:26:39.294373 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.294338 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5l2ht" Jul 06 20:26:39.306212 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:39.306181 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31fefe3_4d3c_4109_aa75_7ca9818d6e4b.slice/crio-4e8a3537a1d88bee1f22738a7de4bac5216f5f76ae3de6e930ba6fad4a0ed94b WatchSource:0}: Error finding container 4e8a3537a1d88bee1f22738a7de4bac5216f5f76ae3de6e930ba6fad4a0ed94b: Status 404 returned error can't find the container with id 4e8a3537a1d88bee1f22738a7de4bac5216f5f76ae3de6e930ba6fad4a0ed94b Jul 06 20:26:39.553783 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:39.553704 2562 scope.go:117] "RemoveContainer" containerID="8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d" Jul 06 20:26:39.554373 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:39.553933 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-hfd54_openshift-console-operator(2d44e299-d22d-49d2-8ab4-b67020497397)\"" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podUID="2d44e299-d22d-49d2-8ab4-b67020497397" Jul 06 20:26:40.208959 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:40.208917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5l2ht" event={"ID":"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b","Type":"ContainerStarted","Data":"4e8a3537a1d88bee1f22738a7de4bac5216f5f76ae3de6e930ba6fad4a0ed94b"} Jul 06 20:26:41.216123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:41.216082 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" event={"ID":"06e67839-19c2-47fc-92d6-10f8a7a517cc","Type":"ContainerStarted","Data":"7d26638af883d847ec85b11e7a5daabb081b2af70d3ab0a3ea8595e8e1e12f89"} Jul 06 20:26:41.217782 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:41.217754 2562 generic.go:356] "Generic (PLEG): container finished" podID="d31fefe3-4d3c-4109-aa75-7ca9818d6e4b" containerID="196dc7d267a871c58f40a904f4184a9ea224ba7ec867a91e35e0b1849e9353b2" exitCode=0 Jul 06 20:26:41.217908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:41.217821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5l2ht" event={"ID":"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b","Type":"ContainerDied","Data":"196dc7d267a871c58f40a904f4184a9ea224ba7ec867a91e35e0b1849e9353b2"} Jul 06 20:26:41.235357 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:41.235301 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cnn6n" podStartSLOduration=1.732068604 podStartE2EDuration="3.235279907s" totalImagesPullingTime="1.503211303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:41.23462733 +0000 UTC m=+143.775683801" watchObservedRunningTime="2026-07-06 20:26:41.235279907 +0000 UTC m=+143.776336343" Jul 06 20:26:42.224107 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:42.224062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5l2ht" event={"ID":"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b","Type":"ContainerStarted","Data":"46ad1408bef4836fcc610b67dbc37fa32209bffc19f0197ec39dd2839925e916"} Jul 06 20:26:42.224522 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:42.224132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5l2ht" event={"ID":"d31fefe3-4d3c-4109-aa75-7ca9818d6e4b","Type":"ContainerStarted","Data":"4451354d8edaf2775721f1f4a659c55313906012850005e994d5e0148d92afa6"} Jul 06 20:26:42.246533 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:42.246452 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5l2ht" podStartSLOduration=3.06171552 podStartE2EDuration="4.246434862s" totalImagesPullingTime="1.115447512s" totalInitContainerRuntime="69.27183ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:42.245653977 +0000 UTC m=+144.786710440" watchObservedRunningTime="2026-07-06 20:26:42.246434862 +0000 UTC m=+144.787491296" Jul 06 20:26:43.582503 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.582469 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5db4f9cd89-28xwp"] Jul 06 20:26:43.585932 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.585916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.588278 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.588254 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 20:26:43.588491 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.588477 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 20:26:43.588491 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.588485 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 20:26:43.588764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.588738 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-khp2k\"" Jul 06 20:26:43.588764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.588757 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 20:26:43.589038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.589022 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 20:26:43.593793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.593772 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 20:26:43.604441 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.604414 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db4f9cd89-28xwp"] Jul 06 20:26:43.709564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-telemeter-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-metrics-client-ca\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdz8\" (UniqueName: \"kubernetes.io/projected/18e53078-35a6-4e84-8051-0713529ec919-kube-api-access-nqdz8\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709937 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-serving-certs-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709937 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.709937 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.709820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-federate-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810369 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810335 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-metrics-client-ca\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810485 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810485 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810485 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdz8\" (UniqueName: \"kubernetes.io/projected/18e53078-35a6-4e84-8051-0713529ec919-kube-api-access-nqdz8\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810485 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-serving-certs-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810485 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810723 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-federate-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.810723 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.810539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-telemeter-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.811299 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.811268 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-metrics-client-ca\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.811413 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.811297 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-serving-certs-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.811413 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.811381 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18e53078-35a6-4e84-8051-0713529ec919-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.813190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.813141 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.813317 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.813238 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-secret-telemeter-client\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.813418 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.813399 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-telemeter-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.813467 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.813417 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/18e53078-35a6-4e84-8051-0713529ec919-federate-client-tls\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.819021 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.818999 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdz8\" (UniqueName: \"kubernetes.io/projected/18e53078-35a6-4e84-8051-0713529ec919-kube-api-access-nqdz8\") pod \"telemeter-client-5db4f9cd89-28xwp\" (UID: \"18e53078-35a6-4e84-8051-0713529ec919\") " pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:43.896134 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:43.896043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" Jul 06 20:26:44.033407 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:44.033349 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db4f9cd89-28xwp"] Jul 06 20:26:44.036279 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:44.036253 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e53078_35a6_4e84_8051_0713529ec919.slice/crio-b3ab6fcf583fb796aafae7d72969efd1742648dfdf7211a70e780a0b1616d33d WatchSource:0}: Error finding container b3ab6fcf583fb796aafae7d72969efd1742648dfdf7211a70e780a0b1616d33d: Status 404 returned error can't find the container with id b3ab6fcf583fb796aafae7d72969efd1742648dfdf7211a70e780a0b1616d33d Jul 06 20:26:44.232470 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:44.232379 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" event={"ID":"18e53078-35a6-4e84-8051-0713529ec919","Type":"ContainerStarted","Data":"b3ab6fcf583fb796aafae7d72969efd1742648dfdf7211a70e780a0b1616d33d"} Jul 06 20:26:47.245814 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:47.245773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" event={"ID":"18e53078-35a6-4e84-8051-0713529ec919","Type":"ContainerStarted","Data":"151c0bc190257c4c9578f3783781916b438be1b6cc8101db33887e823025390f"} Jul 06 20:26:48.076623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:48.076589 2562 patch_prober.go:28] interesting pod/image-registry-6dcb484c46-mmn49 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 20:26:48.076774 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:48.076640 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" podUID="58e0099c-0f37-42d6-9ac3-181ed259799c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:26:48.251301 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:48.251261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" event={"ID":"18e53078-35a6-4e84-8051-0713529ec919","Type":"ContainerStarted","Data":"109684c3a80c7e4928a90b32b541905f0b522feda0577737ab498c91703e213e"} Jul 06 20:26:48.251697 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:48.251349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" event={"ID":"18e53078-35a6-4e84-8051-0713529ec919","Type":"ContainerStarted","Data":"c8c9986a9fc03cdc23f7714504418bc63d0dd577d911958a2e828e952588baa4"} Jul 06 20:26:48.275297 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:48.275244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5db4f9cd89-28xwp" podStartSLOduration=2.075661818 podStartE2EDuration="5.275226136s" totalImagesPullingTime="3.199564318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:48.273658692 +0000 UTC m=+150.814715164" watchObservedRunningTime="2026-07-06 20:26:48.275226136 +0000 UTC m=+150.816282573" Jul 06 20:26:50.150965 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:50.150932 2562 patch_prober.go:28] interesting pod/image-registry-6dcb484c46-mmn49 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 20:26:50.151373 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:50.150989 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" podUID="58e0099c-0f37-42d6-9ac3-181ed259799c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:26:53.553246 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:53.553212 2562 scope.go:117] "RemoveContainer" containerID="8d87d218d401761b30949a8d3007dfee82a4b18c00c575d45a87ca48a2027f9d" Jul 06 20:26:54.275133 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.275109 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:26:54.275323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.275180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" event={"ID":"2d44e299-d22d-49d2-8ab4-b67020497397","Type":"ContainerStarted","Data":"6577b6133ef51567993544079499c9c623dbfbe79274806a4c21fdd2454559ea"} Jul 06 20:26:54.275487 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.275455 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:54.293097 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.293048 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" podStartSLOduration=57.165533344 podStartE2EDuration="59.293032527s" totalImagesPullingTime="2.127499183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:54.291965379 +0000 UTC m=+156.833021865" watchObservedRunningTime="2026-07-06 20:26:54.293032527 +0000 UTC m=+156.834088964" Jul 06 20:26:54.515283 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.515251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-hfd54" Jul 06 20:26:54.719035 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.719001 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-s8xlc"] Jul 06 20:26:54.722282 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.722254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:26:54.724710 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.724682 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 20:26:54.724710 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.724704 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 20:26:54.724884 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.724693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wdbjs\"" Jul 06 20:26:54.736854 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.736831 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-s8xlc"] Jul 06 20:26:54.815229 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.815188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6fn\" (UniqueName: \"kubernetes.io/projected/38829c8e-b2ee-4f20-a4f4-21d637d443b0-kube-api-access-fn6fn\") pod \"downloads-685bfb5f64-s8xlc\" (UID: \"38829c8e-b2ee-4f20-a4f4-21d637d443b0\") " pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:26:54.915801 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.915762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6fn\" (UniqueName: \"kubernetes.io/projected/38829c8e-b2ee-4f20-a4f4-21d637d443b0-kube-api-access-fn6fn\") pod \"downloads-685bfb5f64-s8xlc\" (UID: \"38829c8e-b2ee-4f20-a4f4-21d637d443b0\") " pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:26:54.924868 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:54.924830 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6fn\" (UniqueName: \"kubernetes.io/projected/38829c8e-b2ee-4f20-a4f4-21d637d443b0-kube-api-access-fn6fn\") pod \"downloads-685bfb5f64-s8xlc\" (UID: \"38829c8e-b2ee-4f20-a4f4-21d637d443b0\") " pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:26:55.031986 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:55.031883 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:26:55.164355 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:55.164328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-s8xlc"] Jul 06 20:26:55.166585 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:26:55.166549 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38829c8e_b2ee_4f20_a4f4_21d637d443b0.slice/crio-e51f422c022463da1081f8f6210ccedb7b7f3afac5af767714b0734ecab17663 WatchSource:0}: Error finding container e51f422c022463da1081f8f6210ccedb7b7f3afac5af767714b0734ecab17663: Status 404 returned error can't find the container with id e51f422c022463da1081f8f6210ccedb7b7f3afac5af767714b0734ecab17663 Jul 06 20:26:55.280106 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:55.280067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-s8xlc" event={"ID":"38829c8e-b2ee-4f20-a4f4-21d637d443b0","Type":"ContainerStarted","Data":"e51f422c022463da1081f8f6210ccedb7b7f3afac5af767714b0734ecab17663"} Jul 06 20:26:57.277305 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:26:57.277262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4glv9" podUID="abb79635-08ac-4b40-93d2-f18435f585e8" Jul 06 20:26:57.288612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:57.288588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4glv9" Jul 06 20:26:58.076905 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:58.076872 2562 patch_prober.go:28] interesting pod/image-registry-6dcb484c46-mmn49 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 20:26:58.077109 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:26:58.076928 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" podUID="58e0099c-0f37-42d6-9ac3-181ed259799c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:27:00.150802 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:00.150758 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dcb484c46-mmn49" Jul 06 20:27:02.189891 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:02.185717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:27:02.189891 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:02.188899 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/abb79635-08ac-4b40-93d2-f18435f585e8-metrics-tls\") pod \"dns-default-4glv9\" (UID: \"abb79635-08ac-4b40-93d2-f18435f585e8\") " pod="openshift-dns/dns-default-4glv9" Jul 06 20:27:02.391749 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:02.391712 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvcv5\"" Jul 06 20:27:02.400406 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:02.400362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4glv9" Jul 06 20:27:02.687804 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:02.687778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4glv9"] Jul 06 20:27:02.689700 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:27:02.689667 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb79635_08ac_4b40_93d2_f18435f585e8.slice/crio-a5bfcf85d534f86e5eea727746c2b208bc338e6bf8e30552f90a987eac9cc91e WatchSource:0}: Error finding container a5bfcf85d534f86e5eea727746c2b208bc338e6bf8e30552f90a987eac9cc91e: Status 404 returned error can't find the container with id a5bfcf85d534f86e5eea727746c2b208bc338e6bf8e30552f90a987eac9cc91e Jul 06 20:27:03.316588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:03.316544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4glv9" event={"ID":"abb79635-08ac-4b40-93d2-f18435f585e8","Type":"ContainerStarted","Data":"a5bfcf85d534f86e5eea727746c2b208bc338e6bf8e30552f90a987eac9cc91e"} Jul 06 20:27:05.327128 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:05.327088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4glv9" event={"ID":"abb79635-08ac-4b40-93d2-f18435f585e8","Type":"ContainerStarted","Data":"ca1550614fd0da3e30b8949fcf0a3a0014e45b6b2a4739d4ae13c572033e6b73"} Jul 06 20:27:05.327603 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:05.327161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4glv9" event={"ID":"abb79635-08ac-4b40-93d2-f18435f585e8","Type":"ContainerStarted","Data":"4e5fb6fb4dedb7151a5ed013847cf06787b9448582cb640f2a260910a0732a37"} Jul 06 20:27:05.327603 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:05.327307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4glv9" Jul 06 20:27:05.346189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:05.346107 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4glv9" podStartSLOduration=129.132493575 podStartE2EDuration="2m11.346075922s" totalImagesPullingTime="2.213582347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:05.345901172 +0000 UTC m=+167.886957610" watchObservedRunningTime="2026-07-06 20:27:05.346075922 +0000 UTC m=+167.887132360" Jul 06 20:27:06.141751 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.141718 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:06.155457 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.155424 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:06.155457 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.155446 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.158011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.157973 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:27:06.158011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.158006 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:27:06.158252 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.158012 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:27:06.158252 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.158015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:27:06.158252 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.158020 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:27:06.159708 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.159690 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nr8z5\"" Jul 06 20:27:06.222880 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.222846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.223047 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.222900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr45\" (UniqueName: \"kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.223047 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.222919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.223047 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.222961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.223047 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.223000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.223201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.223051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr45\" (UniqueName: \"kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324852 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.324852 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.324553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.325649 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.325546 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.325649 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.325614 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.325649 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.325631 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.327348 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.327314 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.327712 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.327462 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.332841 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.332817 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr45\" (UniqueName: \"kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45\") pod \"console-76bb989488-zjp2g\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:06.465004 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:06.464915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:08.242605 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:08.242567 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4glv9_abb79635-08ac-4b40-93d2-f18435f585e8/dns/0.log" Jul 06 20:27:08.442612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:08.442580 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4glv9_abb79635-08ac-4b40-93d2-f18435f585e8/kube-rbac-proxy/0.log" Jul 06 20:27:09.642061 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:09.641975 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8np8r_f7083172-62a8-4506-b77e-7112ccde6e98/dns-node-resolver/0.log" Jul 06 20:27:10.242456 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:10.242420 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h97f5_6195ed3c-3037-4c3e-85c5-f381fce16615/serve-healthcheck-canary/0.log" Jul 06 20:27:13.254685 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.254662 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:13.257587 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:27:13.257559 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5f3fa3_e08d_4b54_a68e_67828e94d2b7.slice/crio-8bce5b70ec13eaeae9de0a9acdfbf235de448306c46f68c09db01fad745d6cf4 WatchSource:0}: Error finding container 8bce5b70ec13eaeae9de0a9acdfbf235de448306c46f68c09db01fad745d6cf4: Status 404 returned error can't find the container with id 8bce5b70ec13eaeae9de0a9acdfbf235de448306c46f68c09db01fad745d6cf4 Jul 06 20:27:13.364275 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.364228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-s8xlc" event={"ID":"38829c8e-b2ee-4f20-a4f4-21d637d443b0","Type":"ContainerStarted","Data":"cadd7a08dd1d4086a8d1f9bea07aff8e279f51288ab8f19604d1384e04174455"} Jul 06 20:27:13.364477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.364341 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:27:13.366065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.366029 2562 generic.go:356] "Generic (PLEG): container finished" podID="fbb8cc97-039a-4f28-8c92-aaad7bf9e34f" containerID="75277b0e4c0467098da0f8c68ca9f754730e2f63b8c828866e1ad98f73c6f391" exitCode=0 Jul 06 20:27:13.366212 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.366094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" event={"ID":"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f","Type":"ContainerDied","Data":"75277b0e4c0467098da0f8c68ca9f754730e2f63b8c828866e1ad98f73c6f391"} Jul 06 20:27:13.366212 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.366178 2562 patch_prober.go:28] interesting pod/downloads-685bfb5f64-s8xlc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.22:8080/\": dial tcp 10.133.0.22:8080: connect: connection refused" start-of-body= Jul 06 20:27:13.366340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.366231 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-s8xlc" podUID="38829c8e-b2ee-4f20-a4f4-21d637d443b0" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.22:8080/\": dial tcp 10.133.0.22:8080: connect: connection refused" Jul 06 20:27:13.366497 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.366477 2562 scope.go:117] "RemoveContainer" containerID="75277b0e4c0467098da0f8c68ca9f754730e2f63b8c828866e1ad98f73c6f391" Jul 06 20:27:13.367912 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.367792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb989488-zjp2g" event={"ID":"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7","Type":"ContainerStarted","Data":"8bce5b70ec13eaeae9de0a9acdfbf235de448306c46f68c09db01fad745d6cf4"} Jul 06 20:27:13.384042 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:13.383956 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-s8xlc" podStartSLOduration=1.356513527 podStartE2EDuration="19.383935096s" totalImagesPullingTime="18.027421569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:13.383845281 +0000 UTC m=+175.924901730" watchObservedRunningTime="2026-07-06 20:27:13.383935096 +0000 UTC m=+175.924991533" Jul 06 20:27:14.377858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:14.377279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s9nl5" event={"ID":"fbb8cc97-039a-4f28-8c92-aaad7bf9e34f","Type":"ContainerStarted","Data":"7e64d4994f48eeda6d56b9eb51efad609e51f51d29bdc5c099c01a66b6335669"} Jul 06 20:27:14.389308 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:14.389277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-s8xlc" Jul 06 20:27:15.334473 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.334433 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4glv9" Jul 06 20:27:15.612837 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.612597 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:27:15.631909 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.631185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.632387 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.632326 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:27:15.640600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.640490 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:27:15.714601 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.714805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.714805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.714924 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.714989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.714989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.714975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.715095 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.715051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k97t\" (UniqueName: \"kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815631 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815631 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k97t\" (UniqueName: \"kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.815910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.815833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.816739 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.816699 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.817037 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.817007 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.817793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.817363 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.817793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.817697 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.820460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.820439 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.820552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.820512 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.826054 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.826009 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k97t\" (UniqueName: \"kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t\") pod \"console-6d7b984dfb-2kl64\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:15.949570 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:15.949483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:17.137079 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:17.137025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:27:17.140583 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:27:17.140543 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65533265_2432_479f_9f04_bd5314d48822.slice/crio-088732687bf60de3a03662b0d6fdea6fc2794837644a5dc2bc0bf66bb6ba11fb WatchSource:0}: Error finding container 088732687bf60de3a03662b0d6fdea6fc2794837644a5dc2bc0bf66bb6ba11fb: Status 404 returned error can't find the container with id 088732687bf60de3a03662b0d6fdea6fc2794837644a5dc2bc0bf66bb6ba11fb Jul 06 20:27:17.394529 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:17.394483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7b984dfb-2kl64" event={"ID":"65533265-2432-479f-9f04-bd5314d48822","Type":"ContainerStarted","Data":"088732687bf60de3a03662b0d6fdea6fc2794837644a5dc2bc0bf66bb6ba11fb"} Jul 06 20:27:18.402061 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:18.402021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb989488-zjp2g" event={"ID":"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7","Type":"ContainerStarted","Data":"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8"} Jul 06 20:27:18.405340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:18.405276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7b984dfb-2kl64" event={"ID":"65533265-2432-479f-9f04-bd5314d48822","Type":"ContainerStarted","Data":"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07"} Jul 06 20:27:18.423460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:18.423394 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76bb989488-zjp2g" podStartSLOduration=8.270789393 podStartE2EDuration="12.423374794s" totalImagesPullingTime="4.152585401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:18.421264619 +0000 UTC m=+180.962321060" watchObservedRunningTime="2026-07-06 20:27:18.423374794 +0000 UTC m=+180.964431232" Jul 06 20:27:18.441171 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:18.441099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7b984dfb-2kl64" podStartSLOduration=2.701188418 podStartE2EDuration="3.441072127s" totalImagesPullingTime="739.883709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:18.439577995 +0000 UTC m=+180.980634428" watchObservedRunningTime="2026-07-06 20:27:18.441072127 +0000 UTC m=+180.982128695" Jul 06 20:27:25.950521 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:25.950473 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:25.951118 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:25.950558 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:25.955623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:25.955596 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:26.447730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:26.447698 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:27:26.465555 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:26.465516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:26.465741 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:26.465589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:26.470716 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:26.470690 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:26.518171 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:26.517559 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:27.449301 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:27.449262 2562 generic.go:356] "Generic (PLEG): container finished" podID="292b8f0b-f730-4fdd-a470-73e28ce6602f" containerID="b68bc53efce50f9e01cdb6d25677dd67c08eb77bd01321c9a15367624aeb5a83" exitCode=0 Jul 06 20:27:27.449784 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:27.449340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" event={"ID":"292b8f0b-f730-4fdd-a470-73e28ce6602f","Type":"ContainerDied","Data":"b68bc53efce50f9e01cdb6d25677dd67c08eb77bd01321c9a15367624aeb5a83"} Jul 06 20:27:27.449929 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:27.449913 2562 scope.go:117] "RemoveContainer" containerID="b68bc53efce50f9e01cdb6d25677dd67c08eb77bd01321c9a15367624aeb5a83" Jul 06 20:27:27.453858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:27.453836 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:28.455191 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:28.455137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-sbljx" event={"ID":"292b8f0b-f730-4fdd-a470-73e28ce6602f","Type":"ContainerStarted","Data":"4560153e0ba5231575d7d899a5d7f1d56af1aa11c953dc0765ef9068d5d2de53"} Jul 06 20:27:53.475507 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.475465 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-76bb989488-zjp2g" podUID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" containerName="console" containerID="cri-o://d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8" gracePeriod=15 Jul 06 20:27:53.750619 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.750548 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:53.860907 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.860866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861099 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.860916 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scr45\" (UniqueName: \"kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861099 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.860952 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861099 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.860983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861099 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861032 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861349 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config\") pod \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\" (UID: \"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7\") " Jul 06 20:27:53.861438 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861414 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:27:53.861537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861509 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:27:53.861589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861529 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config" (OuterVolumeSpecName: "console-config") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:27:53.861589 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.861564 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-service-ca\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:53.863486 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.863456 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45" (OuterVolumeSpecName: "kube-api-access-scr45") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "kube-api-access-scr45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:27:53.863486 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.863472 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:27:53.863486 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.863484 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" (UID: "ee5f3fa3-e08d-4b54-a68e-67828e94d2b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:27:53.962319 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.962282 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr45\" (UniqueName: \"kubernetes.io/projected/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-kube-api-access-scr45\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:53.962319 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.962310 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-oauth-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:53.962319 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.962320 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-oauth-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:53.962610 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.962329 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:53.962610 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:53.962338 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7-console-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:27:54.559844 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.559802 2562 generic.go:356] "Generic (PLEG): container finished" podID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" containerID="d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8" exitCode=0 Jul 06 20:27:54.559844 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.559842 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bb989488-zjp2g" Jul 06 20:27:54.560326 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.559873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb989488-zjp2g" event={"ID":"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7","Type":"ContainerDied","Data":"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8"} Jul 06 20:27:54.560326 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.559916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bb989488-zjp2g" event={"ID":"ee5f3fa3-e08d-4b54-a68e-67828e94d2b7","Type":"ContainerDied","Data":"8bce5b70ec13eaeae9de0a9acdfbf235de448306c46f68c09db01fad745d6cf4"} Jul 06 20:27:54.560326 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.559939 2562 scope.go:117] "RemoveContainer" containerID="d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8" Jul 06 20:27:54.570689 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.570674 2562 scope.go:117] "RemoveContainer" containerID="d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8" Jul 06 20:27:54.571134 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:27:54.571113 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8\": container with ID starting with d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8 not found: ID does not exist" containerID="d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8" Jul 06 20:27:54.571259 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.571141 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8"} err="failed to get container status \"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8\": rpc error: code = NotFound desc = could not find container \"d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8\": container with ID starting with d4ed0b5d12d7ef3e541c29ca0043b456a47f479bc6155958b1ed5986ee28c6b8 not found: ID does not exist" Jul 06 20:27:54.583908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.583885 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:54.591697 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.590359 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76bb989488-zjp2g"] Jul 06 20:27:54.719603 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.719424 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:27:54.720327 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.720301 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" containerName="console" Jul 06 20:27:54.720327 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.720327 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" containerName="console" Jul 06 20:27:54.720515 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.720415 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" containerName="console" Jul 06 20:27:54.748879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.748843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:27:54.749034 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.748945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.870966 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.870934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.870978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.870997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.871078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.871107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871334 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.871175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2ll\" (UniqueName: \"kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.871334 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.871206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.971893 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.971840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2ll\" (UniqueName: \"kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.971910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.971942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.971978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.971997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.972028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972100 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.972050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972769 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.972745 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.972811 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.972879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.972837 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.973132 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.973098 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.974699 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.974668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.974776 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.974668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:54.980395 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:54.980375 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2ll\" (UniqueName: \"kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll\") pod \"console-688f6cc798-h2bj7\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:55.058695 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:55.058648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:27:55.221808 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:27:55.221773 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fb3ad8_552c_4f30_8ca7_42e9da2fab06.slice/crio-0d0214349f326fb2ddf2bac0a84b2a75769eab262cdf1f40555b2189dd893fbb WatchSource:0}: Error finding container 0d0214349f326fb2ddf2bac0a84b2a75769eab262cdf1f40555b2189dd893fbb: Status 404 returned error can't find the container with id 0d0214349f326fb2ddf2bac0a84b2a75769eab262cdf1f40555b2189dd893fbb Jul 06 20:27:55.224093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:55.224065 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:27:55.566458 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:55.566419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688f6cc798-h2bj7" event={"ID":"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06","Type":"ContainerStarted","Data":"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da"} Jul 06 20:27:55.566932 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:55.566474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688f6cc798-h2bj7" event={"ID":"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06","Type":"ContainerStarted","Data":"0d0214349f326fb2ddf2bac0a84b2a75769eab262cdf1f40555b2189dd893fbb"} Jul 06 20:27:55.588483 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:55.588393 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-688f6cc798-h2bj7" podStartSLOduration=1.588374662 podStartE2EDuration="1.588374662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:27:55.586953489 +0000 UTC m=+218.128009923" watchObservedRunningTime="2026-07-06 20:27:55.588374662 +0000 UTC m=+218.129431166" Jul 06 20:27:56.557745 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:27:56.557708 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5f3fa3-e08d-4b54-a68e-67828e94d2b7" path="/var/lib/kubelet/pods/ee5f3fa3-e08d-4b54-a68e-67828e94d2b7/volumes" Jul 06 20:28:05.059354 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:05.059314 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:28:05.059834 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:05.059376 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:28:05.064128 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:05.064103 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:28:05.615819 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:05.615788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:28:05.664334 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:05.664296 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:28:30.685894 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.685835 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d7b984dfb-2kl64" podUID="65533265-2432-479f-9f04-bd5314d48822" containerName="console" containerID="cri-o://9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07" gracePeriod=15 Jul 06 20:28:30.930912 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.930886 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:28:30.987108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987012 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987048 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987067 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987133 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987191 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987248 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k97t\" (UniqueName: \"kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t\") pod \"65533265-2432-479f-9f04-bd5314d48822\" (UID: \"65533265-2432-479f-9f04-bd5314d48822\") " Jul 06 20:28:30.987641 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987611 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca" (OuterVolumeSpecName: "service-ca") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:28:30.987703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987626 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:28:30.987703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987647 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:28:30.987703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.987633 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config" (OuterVolumeSpecName: "console-config") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:28:30.989278 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.989254 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:28:30.989344 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.989308 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:28:30.989398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:30.989382 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t" (OuterVolumeSpecName: "kube-api-access-7k97t") pod "65533265-2432-479f-9f04-bd5314d48822" (UID: "65533265-2432-479f-9f04-bd5314d48822"). InnerVolumeSpecName "kube-api-access-7k97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:28:31.088015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.087974 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k97t\" (UniqueName: \"kubernetes.io/projected/65533265-2432-479f-9f04-bd5314d48822-kube-api-access-7k97t\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088007 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-oauth-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088017 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-service-ca\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088028 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-trusted-ca-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088039 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65533265-2432-479f-9f04-bd5314d48822-console-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088048 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-console-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.088324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.088056 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65533265-2432-479f-9f04-bd5314d48822-oauth-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:28:31.719977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.719939 2562 generic.go:356] "Generic (PLEG): container finished" podID="65533265-2432-479f-9f04-bd5314d48822" containerID="9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07" exitCode=0 Jul 06 20:28:31.720508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.719990 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7b984dfb-2kl64" Jul 06 20:28:31.720508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.720019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7b984dfb-2kl64" event={"ID":"65533265-2432-479f-9f04-bd5314d48822","Type":"ContainerDied","Data":"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07"} Jul 06 20:28:31.720508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.720071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7b984dfb-2kl64" event={"ID":"65533265-2432-479f-9f04-bd5314d48822","Type":"ContainerDied","Data":"088732687bf60de3a03662b0d6fdea6fc2794837644a5dc2bc0bf66bb6ba11fb"} Jul 06 20:28:31.720508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.720090 2562 scope.go:117] "RemoveContainer" containerID="9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07" Jul 06 20:28:31.729335 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.729317 2562 scope.go:117] "RemoveContainer" containerID="9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07" Jul 06 20:28:31.729634 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:28:31.729616 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07\": container with ID starting with 9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07 not found: ID does not exist" containerID="9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07" Jul 06 20:28:31.729680 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.729645 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07"} err="failed to get container status \"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07\": rpc error: code = NotFound desc = could not find container \"9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07\": container with ID starting with 9cce269a35538463f2ec953a5d6e1f3695cb51b73ab788255196f628903b7e07 not found: ID does not exist" Jul 06 20:28:31.741615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.741587 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:28:31.742901 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:31.742881 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d7b984dfb-2kl64"] Jul 06 20:28:32.557500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:32.557471 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65533265-2432-479f-9f04-bd5314d48822" path="/var/lib/kubelet/pods/65533265-2432-479f-9f04-bd5314d48822/volumes" Jul 06 20:28:37.244093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.244056 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4"] Jul 06 20:28:37.244566 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.244373 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65533265-2432-479f-9f04-bd5314d48822" containerName="console" Jul 06 20:28:37.244566 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.244385 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="65533265-2432-479f-9f04-bd5314d48822" containerName="console" Jul 06 20:28:37.244566 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.244439 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="65533265-2432-479f-9f04-bd5314d48822" containerName="console" Jul 06 20:28:37.247350 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.247333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.250025 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.250000 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:28:37.250918 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.250897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkq7l\"" Jul 06 20:28:37.250918 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.250909 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:28:37.258403 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.258380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4"] Jul 06 20:28:37.333087 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.333040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbj5\" (UniqueName: \"kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.333296 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.333119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.333296 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.333140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.434093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.434057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.434093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.434094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.434289 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.434169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbj5\" (UniqueName: \"kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.434501 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.434482 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.434549 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.434515 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.443710 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.443677 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbj5\" (UniqueName: \"kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.556572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.556488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:28:37.690697 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.690658 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4"] Jul 06 20:28:37.693858 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:28:37.693827 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2628dab_3fff_4c6e_8221_25ba057e5861.slice/crio-fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6 WatchSource:0}: Error finding container fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6: Status 404 returned error can't find the container with id fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6 Jul 06 20:28:37.746093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:37.746057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" event={"ID":"e2628dab-3fff-4c6e-8221-25ba057e5861","Type":"ContainerStarted","Data":"fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6"} Jul 06 20:28:46.792422 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:46.792382 2562 generic.go:356] "Generic (PLEG): container finished" podID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerID="56e097fc79ba1acefe98665b29d858fb3ba1f478c8f84b934463279f1b696939" exitCode=0 Jul 06 20:28:46.792831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:46.792452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" event={"ID":"e2628dab-3fff-4c6e-8221-25ba057e5861","Type":"ContainerDied","Data":"56e097fc79ba1acefe98665b29d858fb3ba1f478c8f84b934463279f1b696939"} Jul 06 20:28:52.818245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:52.818204 2562 generic.go:356] "Generic (PLEG): container finished" podID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerID="14c44a4982ee033cb4f0899ef1c56834cf63ee93a8b7e9b32fc2efd30eda9d3d" exitCode=0 Jul 06 20:28:52.818245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:28:52.818244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" event={"ID":"e2628dab-3fff-4c6e-8221-25ba057e5861","Type":"ContainerDied","Data":"14c44a4982ee033cb4f0899ef1c56834cf63ee93a8b7e9b32fc2efd30eda9d3d"} Jul 06 20:29:02.863280 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:02.863241 2562 generic.go:356] "Generic (PLEG): container finished" podID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerID="99c966d36a27c352f32195af932fe9164133c8c34b873a9358266a6cb4302e1b" exitCode=0 Jul 06 20:29:02.863663 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:02.863316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" event={"ID":"e2628dab-3fff-4c6e-8221-25ba057e5861","Type":"ContainerDied","Data":"99c966d36a27c352f32195af932fe9164133c8c34b873a9358266a6cb4302e1b"} Jul 06 20:29:04.011393 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.011356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:29:04.161619 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.161531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util\") pod \"e2628dab-3fff-4c6e-8221-25ba057e5861\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " Jul 06 20:29:04.161619 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.161595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scbj5\" (UniqueName: \"kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5\") pod \"e2628dab-3fff-4c6e-8221-25ba057e5861\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " Jul 06 20:29:04.161619 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.161614 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle\") pod \"e2628dab-3fff-4c6e-8221-25ba057e5861\" (UID: \"e2628dab-3fff-4c6e-8221-25ba057e5861\") " Jul 06 20:29:04.162361 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.162323 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle" (OuterVolumeSpecName: "bundle") pod "e2628dab-3fff-4c6e-8221-25ba057e5861" (UID: "e2628dab-3fff-4c6e-8221-25ba057e5861"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:04.163792 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.163767 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5" (OuterVolumeSpecName: "kube-api-access-scbj5") pod "e2628dab-3fff-4c6e-8221-25ba057e5861" (UID: "e2628dab-3fff-4c6e-8221-25ba057e5861"). InnerVolumeSpecName "kube-api-access-scbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:29:04.166781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.166756 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util" (OuterVolumeSpecName: "util") pod "e2628dab-3fff-4c6e-8221-25ba057e5861" (UID: "e2628dab-3fff-4c6e-8221-25ba057e5861"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:04.262478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.262443 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:04.262478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.262473 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbj5\" (UniqueName: \"kubernetes.io/projected/e2628dab-3fff-4c6e-8221-25ba057e5861-kube-api-access-scbj5\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:04.262478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.262482 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2628dab-3fff-4c6e-8221-25ba057e5861-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:04.873362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.873274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" event={"ID":"e2628dab-3fff-4c6e-8221-25ba057e5861","Type":"ContainerDied","Data":"fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6"} Jul 06 20:29:04.873362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.873299 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540chwh4" Jul 06 20:29:04.873362 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:04.873320 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc89c67de4d4c10ee83bccbc330cb61ba56adf46af85a90a802fee2ab7ea2ff6" Jul 06 20:29:10.402802 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.402762 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq"] Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403069 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="extract" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403080 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="extract" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403096 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="pull" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403102 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="pull" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403110 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="util" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403115 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="util" Jul 06 20:29:10.403310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.403187 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2628dab-3fff-4c6e-8221-25ba057e5861" containerName="extract" Jul 06 20:29:10.405390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.405365 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.409524 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.409497 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:29:10.409748 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.409731 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hmsbk\"" Jul 06 20:29:10.409836 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.409813 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 20:29:10.427564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.427536 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq"] Jul 06 20:29:10.515523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.515482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.515523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.515538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxd5\" (UniqueName: \"kubernetes.io/projected/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-kube-api-access-8cxd5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.616523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.616480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.616696 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.616535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxd5\" (UniqueName: \"kubernetes.io/projected/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-kube-api-access-8cxd5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.616890 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.616868 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.625639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.625608 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxd5\" (UniqueName: \"kubernetes.io/projected/37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5-kube-api-access-8cxd5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pxcq\" (UID: \"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.714337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.714257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" Jul 06 20:29:10.866714 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.866667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq"] Jul 06 20:29:10.872575 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:10.872544 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bcc33a_4e5f_4058_9f13_4aeeb2aae6d5.slice/crio-58e9bb93d2aec924143f275de5eb706d20e01daa18c326f254ff4729fdc08a6f WatchSource:0}: Error finding container 58e9bb93d2aec924143f275de5eb706d20e01daa18c326f254ff4729fdc08a6f: Status 404 returned error can't find the container with id 58e9bb93d2aec924143f275de5eb706d20e01daa18c326f254ff4729fdc08a6f Jul 06 20:29:10.900046 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:10.900010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" event={"ID":"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5","Type":"ContainerStarted","Data":"58e9bb93d2aec924143f275de5eb706d20e01daa18c326f254ff4729fdc08a6f"} Jul 06 20:29:13.786803 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.786763 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-587sb"] Jul 06 20:29:13.788945 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.788928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:13.791549 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.791527 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 20:29:13.791672 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.791653 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ml66s\"" Jul 06 20:29:13.792281 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.792266 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 20:29:13.805551 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.805523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-587sb"] Jul 06 20:29:13.917970 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.917932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" event={"ID":"37bcc33a-4e5f-4058-9f13-4aeeb2aae6d5","Type":"ContainerStarted","Data":"5ff82bbe6f4b277ec70c30f3322ff639d3bf35a38a8202c92cc24918201dc097"} Jul 06 20:29:13.943474 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.943420 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pxcq" podStartSLOduration=1.8646515849999998 podStartE2EDuration="3.943402451s" totalImagesPullingTime="2.078750866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:29:13.94209483 +0000 UTC m=+296.483151280" watchObservedRunningTime="2026-07-06 20:29:13.943402451 +0000 UTC m=+296.484458891" Jul 06 20:29:13.945406 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.945377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhgh\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-kube-api-access-hjhgh\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:13.945539 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:13.945487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.046974 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.046890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhgh\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-kube-api-access-hjhgh\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.047124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.047014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.055766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.055733 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.056023 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.056002 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhgh\" (UniqueName: \"kubernetes.io/projected/d879d2dc-6b39-40c0-b353-2ae47099250d-kube-api-access-hjhgh\") pod \"cert-manager-webhook-6486ffd997-587sb\" (UID: \"d879d2dc-6b39-40c0-b353-2ae47099250d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.110733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.110691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:14.240667 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.240640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-587sb"] Jul 06 20:29:14.243119 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:14.243089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd879d2dc_6b39_40c0_b353_2ae47099250d.slice/crio-cd83b1fee11daa218782545f0709ec5c9c4cde6c2d2709ba523140a006e3e470 WatchSource:0}: Error finding container cd83b1fee11daa218782545f0709ec5c9c4cde6c2d2709ba523140a006e3e470: Status 404 returned error can't find the container with id cd83b1fee11daa218782545f0709ec5c9c4cde6c2d2709ba523140a006e3e470 Jul 06 20:29:14.829036 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.829000 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t"] Jul 06 20:29:14.832218 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.832201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:14.834519 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.834496 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkq7l\"" Jul 06 20:29:14.834820 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.834804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:29:14.835456 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.835441 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:29:14.846479 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.846453 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t"] Jul 06 20:29:14.923212 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.923176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" event={"ID":"d879d2dc-6b39-40c0-b353-2ae47099250d","Type":"ContainerStarted","Data":"cd83b1fee11daa218782545f0709ec5c9c4cde6c2d2709ba523140a006e3e470"} Jul 06 20:29:14.955823 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.955782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:14.956001 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.955830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:14.956001 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:14.955926 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l87t\" (UniqueName: \"kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.056543 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.056508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.056738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.056659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.056795 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.056768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l87t\" (UniqueName: \"kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.056931 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.056905 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.057074 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.057049 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.066432 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.066405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l87t\" (UniqueName: \"kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.141976 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.141891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:15.306168 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.306121 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t"] Jul 06 20:29:15.308195 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:15.308122 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18359f49_ede4_4a14_8b60_d6939e850f41.slice/crio-e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10 WatchSource:0}: Error finding container e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10: Status 404 returned error can't find the container with id e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10 Jul 06 20:29:15.930352 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.930311 2562 generic.go:356] "Generic (PLEG): container finished" podID="18359f49-ede4-4a14-8b60-d6939e850f41" containerID="cd90cae86a43246a65ec892c56f95d500a8dfc2726451bc524aae3ab34d30ef2" exitCode=0 Jul 06 20:29:15.930784 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.930376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" event={"ID":"18359f49-ede4-4a14-8b60-d6939e850f41","Type":"ContainerDied","Data":"cd90cae86a43246a65ec892c56f95d500a8dfc2726451bc524aae3ab34d30ef2"} Jul 06 20:29:15.930784 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:15.930434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" event={"ID":"18359f49-ede4-4a14-8b60-d6939e850f41","Type":"ContainerStarted","Data":"e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10"} Jul 06 20:29:16.011989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.011946 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-64d6g"] Jul 06 20:29:16.015240 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.015215 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.018037 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.017851 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hvqz8\"" Jul 06 20:29:16.022801 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.022768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-64d6g"] Jul 06 20:29:16.168051 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.168010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mk84\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-kube-api-access-7mk84\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.168250 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.168207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.269595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.269504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.269595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.269588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mk84\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-kube-api-access-7mk84\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.279738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.279700 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.279910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.279826 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mk84\" (UniqueName: \"kubernetes.io/projected/927d6403-d053-4268-b583-c6ab01fee1d7-kube-api-access-7mk84\") pod \"cert-manager-cainjector-559cdd8c87-64d6g\" (UID: \"927d6403-d053-4268-b583-c6ab01fee1d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:16.330220 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:16.330183 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" Jul 06 20:29:17.131677 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.131646 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-64d6g"] Jul 06 20:29:17.135072 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:17.135033 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod927d6403_d053_4268_b583_c6ab01fee1d7.slice/crio-9b83abd12e56ff76ee4a88c30b683b1e86f0702af9f9887497bf4bebb23c06a2 WatchSource:0}: Error finding container 9b83abd12e56ff76ee4a88c30b683b1e86f0702af9f9887497bf4bebb23c06a2: Status 404 returned error can't find the container with id 9b83abd12e56ff76ee4a88c30b683b1e86f0702af9f9887497bf4bebb23c06a2 Jul 06 20:29:17.942429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.942388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" event={"ID":"927d6403-d053-4268-b583-c6ab01fee1d7","Type":"ContainerStarted","Data":"f84249cfbd7ebef5925d71e13e8b1a77e6ea66a2e43dd825602355450ef999e3"} Jul 06 20:29:17.942623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.942448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" event={"ID":"927d6403-d053-4268-b583-c6ab01fee1d7","Type":"ContainerStarted","Data":"9b83abd12e56ff76ee4a88c30b683b1e86f0702af9f9887497bf4bebb23c06a2"} Jul 06 20:29:17.944307 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.944274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" event={"ID":"d879d2dc-6b39-40c0-b353-2ae47099250d","Type":"ContainerStarted","Data":"2865e04b2569b56ad930aabcd3949cdfee9748c24ca05e2f3696d16e30c7da26"} Jul 06 20:29:17.944454 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.944400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:17.959119 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.958918 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-64d6g" podStartSLOduration=2.958887532 podStartE2EDuration="2.958887532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:29:17.958716784 +0000 UTC m=+300.499773223" watchObservedRunningTime="2026-07-06 20:29:17.958887532 +0000 UTC m=+300.499943969" Jul 06 20:29:17.978338 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:17.978252 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" podStartSLOduration=2.213491751 podStartE2EDuration="4.978210825s" totalImagesPullingTime="2.764719074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:29:17.976614742 +0000 UTC m=+300.517671168" watchObservedRunningTime="2026-07-06 20:29:17.978210825 +0000 UTC m=+300.519267260" Jul 06 20:29:18.408603 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:18.408569 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:29:18.409340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:18.409319 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:29:18.419652 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:18.419634 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 20:29:18.951093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:18.951052 2562 generic.go:356] "Generic (PLEG): container finished" podID="18359f49-ede4-4a14-8b60-d6939e850f41" containerID="f7302d92723d3bb07369c46d8ee6f10a61076da7978d184d51534e8df8c7f33c" exitCode=0 Jul 06 20:29:18.951354 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:18.951137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" event={"ID":"18359f49-ede4-4a14-8b60-d6939e850f41","Type":"ContainerDied","Data":"f7302d92723d3bb07369c46d8ee6f10a61076da7978d184d51534e8df8c7f33c"} Jul 06 20:29:19.957552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:19.957516 2562 generic.go:356] "Generic (PLEG): container finished" podID="18359f49-ede4-4a14-8b60-d6939e850f41" containerID="81f471940e710280d4df8526f449fd4785472b5edfd6f3aad75389b27fee3d5e" exitCode=0 Jul 06 20:29:19.957978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:19.957603 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" event={"ID":"18359f49-ede4-4a14-8b60-d6939e850f41","Type":"ContainerDied","Data":"81f471940e710280d4df8526f449fd4785472b5edfd6f3aad75389b27fee3d5e"} Jul 06 20:29:21.092500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.092476 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:21.113952 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.113921 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util\") pod \"18359f49-ede4-4a14-8b60-d6939e850f41\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " Jul 06 20:29:21.114139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.113985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle\") pod \"18359f49-ede4-4a14-8b60-d6939e850f41\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " Jul 06 20:29:21.114139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.114030 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l87t\" (UniqueName: \"kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t\") pod \"18359f49-ede4-4a14-8b60-d6939e850f41\" (UID: \"18359f49-ede4-4a14-8b60-d6939e850f41\") " Jul 06 20:29:21.114458 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.114432 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle" (OuterVolumeSpecName: "bundle") pod "18359f49-ede4-4a14-8b60-d6939e850f41" (UID: "18359f49-ede4-4a14-8b60-d6939e850f41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:21.116395 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.116368 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t" (OuterVolumeSpecName: "kube-api-access-9l87t") pod "18359f49-ede4-4a14-8b60-d6939e850f41" (UID: "18359f49-ede4-4a14-8b60-d6939e850f41"). InnerVolumeSpecName "kube-api-access-9l87t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:29:21.119172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.119132 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util" (OuterVolumeSpecName: "util") pod "18359f49-ede4-4a14-8b60-d6939e850f41" (UID: "18359f49-ede4-4a14-8b60-d6939e850f41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:21.215721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.215621 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l87t\" (UniqueName: \"kubernetes.io/projected/18359f49-ede4-4a14-8b60-d6939e850f41-kube-api-access-9l87t\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:21.215721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.215655 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:21.215721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.215668 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18359f49-ede4-4a14-8b60-d6939e850f41-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:21.968550 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.968515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" event={"ID":"18359f49-ede4-4a14-8b60-d6939e850f41","Type":"ContainerDied","Data":"e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10"} Jul 06 20:29:21.968550 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.968548 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9x74t" Jul 06 20:29:21.968779 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:21.968567 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1049763d58a0dd52c139031de5f683ffd3738b2b762a19ba899f45ca841de10" Jul 06 20:29:23.953499 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:23.953468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-587sb" Jul 06 20:29:26.294773 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.294730 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-m2p9z"] Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295207 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="extract" Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295226 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="extract" Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295239 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="util" Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295248 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="util" Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295277 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="pull" Jul 06 20:29:26.295284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295285 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="pull" Jul 06 20:29:26.295588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.295373 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="18359f49-ede4-4a14-8b60-d6939e850f41" containerName="extract" Jul 06 20:29:26.297435 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.297415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.299658 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.299634 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x88mc\"" Jul 06 20:29:26.309245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.309221 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-m2p9z"] Jul 06 20:29:26.355556 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.355522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-bound-sa-token\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.355730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.355584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjh9t\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-kube-api-access-cjh9t\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.456936 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.456893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-bound-sa-token\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.457118 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.456956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjh9t\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-kube-api-access-cjh9t\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.465837 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.465810 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-bound-sa-token\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.465994 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.465958 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjh9t\" (UniqueName: \"kubernetes.io/projected/7bfbf738-5b2d-4750-8b09-7a37c314855f-kube-api-access-cjh9t\") pod \"cert-manager-55b4d954d5-m2p9z\" (UID: \"7bfbf738-5b2d-4750-8b09-7a37c314855f\") " pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.606650 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.606615 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-m2p9z" Jul 06 20:29:26.739564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.739526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-m2p9z"] Jul 06 20:29:26.743991 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:26.743958 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bfbf738_5b2d_4750_8b09_7a37c314855f.slice/crio-cb8b04df337367e5e76bf18f41d6f1c95cdece3aa6d9e2080665461fe1fccee7 WatchSource:0}: Error finding container cb8b04df337367e5e76bf18f41d6f1c95cdece3aa6d9e2080665461fe1fccee7: Status 404 returned error can't find the container with id cb8b04df337367e5e76bf18f41d6f1c95cdece3aa6d9e2080665461fe1fccee7 Jul 06 20:29:26.998805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.998708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-m2p9z" event={"ID":"7bfbf738-5b2d-4750-8b09-7a37c314855f","Type":"ContainerStarted","Data":"c287f7c833eb1cde6a57799e4ad2481f7df7fbb715e9411ca7ccee44651296ec"} Jul 06 20:29:26.998805 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:26.998752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-m2p9z" event={"ID":"7bfbf738-5b2d-4750-8b09-7a37c314855f","Type":"ContainerStarted","Data":"cb8b04df337367e5e76bf18f41d6f1c95cdece3aa6d9e2080665461fe1fccee7"} Jul 06 20:29:27.018349 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:27.018293 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-m2p9z" podStartSLOduration=1.018276033 podStartE2EDuration="1.018276033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:29:27.017429158 +0000 UTC m=+309.558485602" watchObservedRunningTime="2026-07-06 20:29:27.018276033 +0000 UTC m=+309.559332470" Jul 06 20:29:34.924832 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.924792 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6"] Jul 06 20:29:34.973550 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.973508 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6"] Jul 06 20:29:34.973750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.973651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:34.976574 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.976545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jkq7l\"" Jul 06 20:29:34.976726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.976544 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:29:34.976726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:34.976589 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:29:35.030250 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.030215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.030437 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.030275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4nj\" (UniqueName: \"kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.030437 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.030296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.131139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.131105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4nj\" (UniqueName: \"kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.131139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.131142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.131382 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.131217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.131555 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.131538 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.131588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.131561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.147036 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.147010 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4nj\" (UniqueName: \"kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.286191 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.286063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:35.446366 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:35.446338 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6"] Jul 06 20:29:35.448460 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:35.448432 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d27a842_ae93_42f4_a195_19994fbec892.slice/crio-9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750 WatchSource:0}: Error finding container 9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750: Status 404 returned error can't find the container with id 9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750 Jul 06 20:29:36.040692 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:36.040654 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d27a842-ae93-42f4-a195-19994fbec892" containerID="0c5fb422b5c0a9c8a8547afea94a3a08eedb3c0d98fad64f8bd6ae9dc5707f8b" exitCode=0 Jul 06 20:29:36.041106 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:36.040741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" event={"ID":"9d27a842-ae93-42f4-a195-19994fbec892","Type":"ContainerDied","Data":"0c5fb422b5c0a9c8a8547afea94a3a08eedb3c0d98fad64f8bd6ae9dc5707f8b"} Jul 06 20:29:36.041106 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:36.040792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" event={"ID":"9d27a842-ae93-42f4-a195-19994fbec892","Type":"ContainerStarted","Data":"9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750"} Jul 06 20:29:36.041292 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:36.041277 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:29:38.051752 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:38.051712 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d27a842-ae93-42f4-a195-19994fbec892" containerID="e97286de6fff19172649cd12e524b9c963dcea45c7ca92bc18d807e296033ade" exitCode=0 Jul 06 20:29:38.052202 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:38.051784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" event={"ID":"9d27a842-ae93-42f4-a195-19994fbec892","Type":"ContainerDied","Data":"e97286de6fff19172649cd12e524b9c963dcea45c7ca92bc18d807e296033ade"} Jul 06 20:29:39.058140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:39.058097 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d27a842-ae93-42f4-a195-19994fbec892" containerID="22cdc2853c86df5ce57524525f3ae98226454223c1e595107786e32ee6115012" exitCode=0 Jul 06 20:29:39.058140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:39.058143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" event={"ID":"9d27a842-ae93-42f4-a195-19994fbec892","Type":"ContainerDied","Data":"22cdc2853c86df5ce57524525f3ae98226454223c1e595107786e32ee6115012"} Jul 06 20:29:40.188674 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.188649 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:40.276988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.276954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc4nj\" (UniqueName: \"kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj\") pod \"9d27a842-ae93-42f4-a195-19994fbec892\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " Jul 06 20:29:40.277235 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.277079 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle\") pod \"9d27a842-ae93-42f4-a195-19994fbec892\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " Jul 06 20:29:40.277235 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.277111 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util\") pod \"9d27a842-ae93-42f4-a195-19994fbec892\" (UID: \"9d27a842-ae93-42f4-a195-19994fbec892\") " Jul 06 20:29:40.277818 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.277780 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle" (OuterVolumeSpecName: "bundle") pod "9d27a842-ae93-42f4-a195-19994fbec892" (UID: "9d27a842-ae93-42f4-a195-19994fbec892"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:40.279161 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.279119 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj" (OuterVolumeSpecName: "kube-api-access-tc4nj") pod "9d27a842-ae93-42f4-a195-19994fbec892" (UID: "9d27a842-ae93-42f4-a195-19994fbec892"). InnerVolumeSpecName "kube-api-access-tc4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:29:40.282761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.282725 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util" (OuterVolumeSpecName: "util") pod "9d27a842-ae93-42f4-a195-19994fbec892" (UID: "9d27a842-ae93-42f4-a195-19994fbec892"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:29:40.378050 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.377962 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:40.378050 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.377994 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d27a842-ae93-42f4-a195-19994fbec892-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:40.378050 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:40.378003 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4nj\" (UniqueName: \"kubernetes.io/projected/9d27a842-ae93-42f4-a195-19994fbec892-kube-api-access-tc4nj\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:29:41.069691 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:41.069659 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" Jul 06 20:29:41.069869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:41.069693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af295p6" event={"ID":"9d27a842-ae93-42f4-a195-19994fbec892","Type":"ContainerDied","Data":"9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750"} Jul 06 20:29:41.069869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:41.069739 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc5276d9019c00270cc3d8d57e7264d4cf2917fe730667dadb773c9eccea750" Jul 06 20:29:52.593890 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.593857 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b"] Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594226 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="pull" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594273 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="pull" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594283 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="util" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594289 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="util" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594301 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="extract" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="extract" Jul 06 20:29:52.594390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.594371 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d27a842-ae93-42f4-a195-19994fbec892" containerName="extract" Jul 06 20:29:52.596861 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.596843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.609034 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.609016 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 20:29:52.609385 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.609368 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 20:29:52.610003 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.609984 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mc6vc\"" Jul 06 20:29:52.610227 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.610208 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 20:29:52.617047 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.617023 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b"] Jul 06 20:29:52.654596 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.654565 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 20:29:52.688584 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.688546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgqv\" (UniqueName: \"kubernetes.io/projected/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-kube-api-access-vdgqv\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.688887 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.688852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.689048 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.689033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.689222 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.689193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-tmp\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.689423 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.689407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-manifests\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790160 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgqv\" (UniqueName: \"kubernetes.io/projected/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-kube-api-access-vdgqv\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790338 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790380 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790424 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-tmp\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790500 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-manifests\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.790864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.790840 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-manifests\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.792853 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.792817 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-tmp\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.793023 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.793003 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.793134 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.793110 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.825531 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.825502 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgqv\" (UniqueName: \"kubernetes.io/projected/b0812b4d-daff-4ea7-8bf3-236ab9611fc6-kube-api-access-vdgqv\") pod \"opendatahub-operator-controller-manager-67f68bd595-brq2b\" (UID: \"b0812b4d-daff-4ea7-8bf3-236ab9611fc6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:52.907282 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:52.907201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:29:53.074142 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:53.074107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b"] Jul 06 20:29:53.077645 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:29:53.077616 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0812b4d_daff_4ea7_8bf3_236ab9611fc6.slice/crio-1dddf94af3056180c857449087e29f51da0b39333d4db161d2a7082bcbd187f6 WatchSource:0}: Error finding container 1dddf94af3056180c857449087e29f51da0b39333d4db161d2a7082bcbd187f6: Status 404 returned error can't find the container with id 1dddf94af3056180c857449087e29f51da0b39333d4db161d2a7082bcbd187f6 Jul 06 20:29:53.125591 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:53.125549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" event={"ID":"b0812b4d-daff-4ea7-8bf3-236ab9611fc6","Type":"ContainerStarted","Data":"1dddf94af3056180c857449087e29f51da0b39333d4db161d2a7082bcbd187f6"} Jul 06 20:29:57.146320 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:57.146284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" event={"ID":"b0812b4d-daff-4ea7-8bf3-236ab9611fc6","Type":"ContainerStarted","Data":"b00a86a3dc254dccc159076ac51875dbf7a6d6a6db3a22732ca7b8943e6b26a5"} Jul 06 20:29:58.152165 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:58.152118 2562 generic.go:356] "Generic (PLEG): container finished" podID="b0812b4d-daff-4ea7-8bf3-236ab9611fc6" containerID="b00a86a3dc254dccc159076ac51875dbf7a6d6a6db3a22732ca7b8943e6b26a5" exitCode=0 Jul 06 20:29:58.152576 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:58.152186 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" event={"ID":"b0812b4d-daff-4ea7-8bf3-236ab9611fc6","Type":"ContainerDied","Data":"b00a86a3dc254dccc159076ac51875dbf7a6d6a6db3a22732ca7b8943e6b26a5"} Jul 06 20:29:59.159012 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:59.158968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" event={"ID":"b0812b4d-daff-4ea7-8bf3-236ab9611fc6","Type":"ContainerStarted","Data":"9a618c7f4eb90dda345b8bf6ed93a45340d5ae8c8b98972ab25753ce074501a7"} Jul 06 20:29:59.159473 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:29:59.159082 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:30:10.165776 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:10.165695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" Jul 06 20:30:10.190204 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:10.190127 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f68bd595-brq2b" podStartSLOduration=13.793080776 podStartE2EDuration="18.19010391s" totalImagesPullingTime="3.362715076s" totalInitContainerRuntime="1.034308058s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:29:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:29:59.190165988 +0000 UTC m=+341.731222425" watchObservedRunningTime="2026-07-06 20:30:10.19010391 +0000 UTC m=+352.731160350" Jul 06 20:30:18.640922 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.640887 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f"] Jul 06 20:30:18.645552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.645533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.650515 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.650485 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 20:30:18.650647 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.650491 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 20:30:18.650647 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.650551 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 20:30:18.650761 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.650748 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 20:30:18.656978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.656956 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f"] Jul 06 20:30:18.715423 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.715383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb612596-dbed-411b-8062-510383d80c94-tls-certs\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.715608 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.715431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66v46\" (UniqueName: \"kubernetes.io/projected/eb612596-dbed-411b-8062-510383d80c94-kube-api-access-66v46\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.715608 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.715493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb612596-dbed-411b-8062-510383d80c94-tmp\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.810932 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.810894 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qwmwr"] Jul 06 20:30:18.814189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.814173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.815867 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.815842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb612596-dbed-411b-8062-510383d80c94-tmp\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.815976 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.815879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkn7l\" (UniqueName: \"kubernetes.io/projected/3b04bd76-df05-420e-b00d-1625dea722cb-kube-api-access-xkn7l\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.815976 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.815918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb612596-dbed-411b-8062-510383d80c94-tls-certs\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.816044 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.815972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66v46\" (UniqueName: \"kubernetes.io/projected/eb612596-dbed-411b-8062-510383d80c94-kube-api-access-66v46\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.816044 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.816014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.816568 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.816547 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 20:30:18.816754 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.816734 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wstk8\"" Jul 06 20:30:18.818305 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.818273 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb612596-dbed-411b-8062-510383d80c94-tmp\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.818451 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.818437 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb612596-dbed-411b-8062-510383d80c94-tls-certs\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.825307 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.825283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qwmwr"] Jul 06 20:30:18.838208 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.838181 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66v46\" (UniqueName: \"kubernetes.io/projected/eb612596-dbed-411b-8062-510383d80c94-kube-api-access-66v46\") pod \"kube-auth-proxy-cd668b4d4-5v95f\" (UID: \"eb612596-dbed-411b-8062-510383d80c94\") " pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:18.916684 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.916595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.916684 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.916653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkn7l\" (UniqueName: \"kubernetes.io/projected/3b04bd76-df05-420e-b00d-1625dea722cb-kube-api-access-xkn7l\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.916861 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:30:18.916747 2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 20:30:18.916861 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:30:18.916810 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert podName:3b04bd76-df05-420e-b00d-1625dea722cb nodeName:}" failed. No retries permitted until 2026-07-06 20:30:19.416792226 +0000 UTC m=+361.957848650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert") pod "odh-model-controller-7998c67dc4-qwmwr" (UID: "3b04bd76-df05-420e-b00d-1625dea722cb") : secret "odh-model-controller-webhook-cert" not found Jul 06 20:30:18.925817 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.925794 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkn7l\" (UniqueName: \"kubernetes.io/projected/3b04bd76-df05-420e-b00d-1625dea722cb-kube-api-access-xkn7l\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:18.955523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:18.955491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" Jul 06 20:30:19.095433 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.095404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f"] Jul 06 20:30:19.097779 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:19.097747 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb612596_dbed_411b_8062_510383d80c94.slice/crio-efc06d5fe523f701b856e4e166a0f0bc9b328b7e29200e83b3c091f7d2be0e8d WatchSource:0}: Error finding container efc06d5fe523f701b856e4e166a0f0bc9b328b7e29200e83b3c091f7d2be0e8d: Status 404 returned error can't find the container with id efc06d5fe523f701b856e4e166a0f0bc9b328b7e29200e83b3c091f7d2be0e8d Jul 06 20:30:19.266016 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.265921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" event={"ID":"eb612596-dbed-411b-8062-510383d80c94","Type":"ContainerStarted","Data":"efc06d5fe523f701b856e4e166a0f0bc9b328b7e29200e83b3c091f7d2be0e8d"} Jul 06 20:30:19.368586 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.368552 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vwcmz"] Jul 06 20:30:19.373141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.373118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.376096 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.376073 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 20:30:19.376253 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.376227 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-lhnzr\"" Jul 06 20:30:19.388106 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.388084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vwcmz"] Jul 06 20:30:19.420035 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.420002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9chs\" (UniqueName: \"kubernetes.io/projected/441b480e-8082-482a-a928-e1260cde0e68-kube-api-access-n9chs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.420200 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.420078 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:19.420200 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.420109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441b480e-8082-482a-a928-e1260cde0e68-tls-certs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.422506 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.422483 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b04bd76-df05-420e-b00d-1625dea722cb-cert\") pod \"odh-model-controller-7998c67dc4-qwmwr\" (UID: \"3b04bd76-df05-420e-b00d-1625dea722cb\") " pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:19.452472 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.452433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:19.521064 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.520991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441b480e-8082-482a-a928-e1260cde0e68-tls-certs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.521064 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.521036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9chs\" (UniqueName: \"kubernetes.io/projected/441b480e-8082-482a-a928-e1260cde0e68-kube-api-access-n9chs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.523559 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.523534 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441b480e-8082-482a-a928-e1260cde0e68-tls-certs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.530480 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.530455 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9chs\" (UniqueName: \"kubernetes.io/projected/441b480e-8082-482a-a928-e1260cde0e68-kube-api-access-n9chs\") pod \"model-serving-api-54d89bd5fc-vwcmz\" (UID: \"441b480e-8082-482a-a928-e1260cde0e68\") " pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.607427 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.607397 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qwmwr"] Jul 06 20:30:19.608684 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:19.608657 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b04bd76_df05_420e_b00d_1625dea722cb.slice/crio-0ce9b61724067ee7cd0626af5bd54ab08606cc9b64473cf108958d04ca6f7932 WatchSource:0}: Error finding container 0ce9b61724067ee7cd0626af5bd54ab08606cc9b64473cf108958d04ca6f7932: Status 404 returned error can't find the container with id 0ce9b61724067ee7cd0626af5bd54ab08606cc9b64473cf108958d04ca6f7932 Jul 06 20:30:19.683473 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.683434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:19.852332 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:19.852302 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vwcmz"] Jul 06 20:30:19.854442 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:19.854404 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441b480e_8082_482a_a928_e1260cde0e68.slice/crio-d9cbce7c8ef7e2225ffa07ab574a16be6a95af73fcde349d23cf8a27a1decaec WatchSource:0}: Error finding container d9cbce7c8ef7e2225ffa07ab574a16be6a95af73fcde349d23cf8a27a1decaec: Status 404 returned error can't find the container with id d9cbce7c8ef7e2225ffa07ab574a16be6a95af73fcde349d23cf8a27a1decaec Jul 06 20:30:20.280194 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.280076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" event={"ID":"441b480e-8082-482a-a928-e1260cde0e68","Type":"ContainerStarted","Data":"d9cbce7c8ef7e2225ffa07ab574a16be6a95af73fcde349d23cf8a27a1decaec"} Jul 06 20:30:20.281717 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.281684 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" event={"ID":"3b04bd76-df05-420e-b00d-1625dea722cb","Type":"ContainerStarted","Data":"0ce9b61724067ee7cd0626af5bd54ab08606cc9b64473cf108958d04ca6f7932"} Jul 06 20:30:20.301163 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.301002 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:20.304887 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.304859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.307540 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.307514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 20:30:20.307699 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.307636 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 20:30:20.307699 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.307671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 20:30:20.307699 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.307681 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-72rtj\"" Jul 06 20:30:20.307873 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.307774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 20:30:20.317482 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.317457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:20.329714 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.329875 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.329875 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.329875 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.329875 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.330130 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptlb\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.330130 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.329925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431181 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431381 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431469 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431469 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.431597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.431584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptlb\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.432595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.432566 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.434229 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.434205 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.434518 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.434497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.434626 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.434605 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.435301 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.435283 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.443432 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.443402 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.444090 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.444067 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptlb\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb\") pod \"istiod-openshift-gateway-7574459656-9mj7v\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.619960 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.619916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:20.817767 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:20.817737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:20.819354 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:20.819324 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede4e587_d0ca_4de4_8218_e62a5951d87f.slice/crio-48852b31b40afe74b69268a81ba7b32bda18652aacf71f55f18d95cf301f3f70 WatchSource:0}: Error finding container 48852b31b40afe74b69268a81ba7b32bda18652aacf71f55f18d95cf301f3f70: Status 404 returned error can't find the container with id 48852b31b40afe74b69268a81ba7b32bda18652aacf71f55f18d95cf301f3f70 Jul 06 20:30:21.288483 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:21.288449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" event={"ID":"ede4e587-d0ca-4de4-8218-e62a5951d87f","Type":"ContainerStarted","Data":"48852b31b40afe74b69268a81ba7b32bda18652aacf71f55f18d95cf301f3f70"} Jul 06 20:30:24.887660 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:24.887613 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:30:24.887996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:24.887697 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:30:25.315005 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.314967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" event={"ID":"eb612596-dbed-411b-8062-510383d80c94","Type":"ContainerStarted","Data":"db6e00fec71cea671962d0726fd9bd85ef1388fc57cd61a756b6667d15aa2ff9"} Jul 06 20:30:25.316860 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.316817 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" event={"ID":"441b480e-8082-482a-a928-e1260cde0e68","Type":"ContainerStarted","Data":"4e044d93734bdccf23ac783b9d07641369402cd7e79669bbb2fdfba17b471774"} Jul 06 20:30:25.317010 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.316924 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:25.318633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.318595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" event={"ID":"ede4e587-d0ca-4de4-8218-e62a5951d87f","Type":"ContainerStarted","Data":"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c"} Jul 06 20:30:25.318857 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.318835 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:25.320695 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.320669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" event={"ID":"3b04bd76-df05-420e-b00d-1625dea722cb","Type":"ContainerStarted","Data":"625fafed6cc0d32963712d21f764b30fa4f27b23f98eaa9b156f5c63a1038141"} Jul 06 20:30:25.320829 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.320739 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9mj7v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:30:25.320829 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.320770 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:25.320829 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.320803 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:30:25.337981 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.337932 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cd668b4d4-5v95f" podStartSLOduration=1.615811517 podStartE2EDuration="7.337916628s" totalImagesPullingTime="5.722105111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:25.336080213 +0000 UTC m=+367.877136641" watchObservedRunningTime="2026-07-06 20:30:25.337916628 +0000 UTC m=+367.878973066" Jul 06 20:30:25.356728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.356671 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" podStartSLOduration=2.276705482 podStartE2EDuration="6.356656179s" totalImagesPullingTime="4.079950697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:25.354638916 +0000 UTC m=+367.895695353" watchObservedRunningTime="2026-07-06 20:30:25.356656179 +0000 UTC m=+367.897712617" Jul 06 20:30:25.393360 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.387443 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" podStartSLOduration=1.322528694 podStartE2EDuration="5.387422091s" totalImagesPullingTime="4.064893397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:25.385507678 +0000 UTC m=+367.926564311" watchObservedRunningTime="2026-07-06 20:30:25.387422091 +0000 UTC m=+367.928478529" Jul 06 20:30:25.408497 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:25.408438 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" podStartSLOduration=2.187396235 podStartE2EDuration="7.408420637s" totalImagesPullingTime="5.221024402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:25.407386775 +0000 UTC m=+367.948443212" watchObservedRunningTime="2026-07-06 20:30:25.408420637 +0000 UTC m=+367.949477075" Jul 06 20:30:26.326390 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.326358 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9mj7v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:30:26.326771 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.326419 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:30:26.792860 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.792774 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29"] Jul 06 20:30:26.796430 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.796407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.802268 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.802241 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7qkbz\"" Jul 06 20:30:26.835578 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.835542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29"] Jul 06 20:30:26.897523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qc28\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-kube-api-access-9qc28\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897903 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897726 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897903 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.897903 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.897800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qc28\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-kube-api-access-9qc28\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.998978 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.998983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.999199 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.999381 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.999418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.999473 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:26.999613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:26.999550 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.001191 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.001170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.001512 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.001490 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.015272 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.015230 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.015437 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.015409 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qc28\" (UniqueName: \"kubernetes.io/projected/e5d89105-22b2-4326-b631-6ae10d8f7c8c-kube-api-access-9qc28\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ghw29\" (UID: \"e5d89105-22b2-4326-b631-6ae10d8f7c8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.107325 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.107283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:27.319888 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.319860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29"] Jul 06 20:30:27.320833 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:27.320807 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d89105_22b2_4326_b631_6ae10d8f7c8c.slice/crio-d33177d22f501c1fb58299728aa3fbf5a715d623f80e186a324d3d52a52721ea WatchSource:0}: Error finding container d33177d22f501c1fb58299728aa3fbf5a715d623f80e186a324d3d52a52721ea: Status 404 returned error can't find the container with id d33177d22f501c1fb58299728aa3fbf5a715d623f80e186a324d3d52a52721ea Jul 06 20:30:27.332551 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:27.332523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" event={"ID":"e5d89105-22b2-4326-b631-6ae10d8f7c8c","Type":"ContainerStarted","Data":"d33177d22f501c1fb58299728aa3fbf5a715d623f80e186a324d3d52a52721ea"} Jul 06 20:30:29.327521 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:29.327486 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:29.879683 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:29.879638 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:30:29.879757 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:29.879734 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:30:29.879793 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:29.879762 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:30:30.324633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.324583 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:30.324831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.324806 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" containerID="cri-o://6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c" gracePeriod=30 Jul 06 20:30:30.354175 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.354116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" event={"ID":"e5d89105-22b2-4326-b631-6ae10d8f7c8c","Type":"ContainerStarted","Data":"70b22c381436b44e01d4e7a304caa5fb614298a25673d22282ccb4cffecc666b"} Jul 06 20:30:30.456740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.456671 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" podStartSLOduration=1.9003647369999999 podStartE2EDuration="4.456652911s" totalImagesPullingTime="2.556288174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:30.455866812 +0000 UTC m=+372.996923250" watchObservedRunningTime="2026-07-06 20:30:30.456652911 +0000 UTC m=+372.997709347" Jul 06 20:30:30.572387 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.572361 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:30.635864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.635864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635807 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.635864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.636124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635910 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.636124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.636124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.635980 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.636124 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.636014 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pptlb\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb\") pod \"ede4e587-d0ca-4de4-8218-e62a5951d87f\" (UID: \"ede4e587-d0ca-4de4-8218-e62a5951d87f\") " Jul 06 20:30:30.636850 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.636816 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:30:30.638917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.638742 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs" (OuterVolumeSpecName: "local-certs") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:30:30.638917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.638814 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:30:30.638917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.638875 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb" (OuterVolumeSpecName: "kube-api-access-pptlb") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "kube-api-access-pptlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:30.638917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.638872 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:30:30.639123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.638995 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:30.639123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.639016 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts" (OuterVolumeSpecName: "cacerts") pod "ede4e587-d0ca-4de4-8218-e62a5951d87f" (UID: "ede4e587-d0ca-4de4-8218-e62a5951d87f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:30:30.737337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737283 2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede4e587-d0ca-4de4-8218-e62a5951d87f-local-certs\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737331 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-ca-configmap\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737347 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-token\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737361 2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-kubeconfig\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737373 2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-cacerts\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737384 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptlb\" (UniqueName: \"kubernetes.io/projected/ede4e587-d0ca-4de4-8218-e62a5951d87f-kube-api-access-pptlb\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:30.737600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:30.737396 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ede4e587-d0ca-4de4-8218-e62a5951d87f-istio-csr-dns-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:30:31.107878 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.107844 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:31.113210 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.113188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:31.347291 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.347256 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp"] Jul 06 20:30:31.347623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.347611 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" Jul 06 20:30:31.347682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.347625 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" Jul 06 20:30:31.347718 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.347699 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerName="discovery" Jul 06 20:30:31.351023 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.351001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.353460 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.353434 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgn22"] Jul 06 20:30:31.356585 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.356564 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.357130 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.357112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 20:30:31.357315 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.357300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lz2hb\"" Jul 06 20:30:31.359204 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.359037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-q8pjc\"" Jul 06 20:30:31.361894 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.361868 2562 generic.go:356] "Generic (PLEG): container finished" podID="ede4e587-d0ca-4de4-8218-e62a5951d87f" containerID="6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c" exitCode=0 Jul 06 20:30:31.362002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.361900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" event={"ID":"ede4e587-d0ca-4de4-8218-e62a5951d87f","Type":"ContainerDied","Data":"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c"} Jul 06 20:30:31.362002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.361931 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" Jul 06 20:30:31.362002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.361951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v" event={"ID":"ede4e587-d0ca-4de4-8218-e62a5951d87f","Type":"ContainerDied","Data":"48852b31b40afe74b69268a81ba7b32bda18652aacf71f55f18d95cf301f3f70"} Jul 06 20:30:31.362002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.361976 2562 scope.go:117] "RemoveContainer" containerID="6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c" Jul 06 20:30:31.362474 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.362393 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:31.363813 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.363570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ghw29" Jul 06 20:30:31.364699 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.364680 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 20:30:31.373693 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.373676 2562 scope.go:117] "RemoveContainer" containerID="6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c" Jul 06 20:30:31.373976 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:30:31.373951 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c\": container with ID starting with 6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c not found: ID does not exist" containerID="6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c" Jul 06 20:30:31.374066 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.373982 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c"} err="failed to get container status \"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c\": rpc error: code = NotFound desc = could not find container \"6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c\": container with ID starting with 6ee2e9bcda963aea28384df38989b74bb059aabe8c7a5be9e4ac981ad445121c not found: ID does not exist" Jul 06 20:30:31.392731 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.392696 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgn22"] Jul 06 20:30:31.394650 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.394625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp"] Jul 06 20:30:31.442956 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.442923 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:31.443781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.443762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10660b83-85b5-4291-8cd5-ef588ed0898a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.443893 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.443789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b9a371-da48-4c41-99b3-c77decdd73e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.443940 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.443895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w6d\" (UniqueName: \"kubernetes.io/projected/10660b83-85b5-4291-8cd5-ef588ed0898a-kube-api-access-z6w6d\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.444020 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.444001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lt5\" (UniqueName: \"kubernetes.io/projected/b1b9a371-da48-4c41-99b3-c77decdd73e6-kube-api-access-48lt5\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.448240 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.448218 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9mj7v"] Jul 06 20:30:31.544580 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.544544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10660b83-85b5-4291-8cd5-ef588ed0898a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.544580 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.544583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b9a371-da48-4c41-99b3-c77decdd73e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.544838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.544646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w6d\" (UniqueName: \"kubernetes.io/projected/10660b83-85b5-4291-8cd5-ef588ed0898a-kube-api-access-z6w6d\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.544838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.544665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lt5\" (UniqueName: \"kubernetes.io/projected/b1b9a371-da48-4c41-99b3-c77decdd73e6-kube-api-access-48lt5\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.547029 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.547003 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10660b83-85b5-4291-8cd5-ef588ed0898a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.547133 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.547115 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b9a371-da48-4c41-99b3-c77decdd73e6-cert\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.556745 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.556716 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lt5\" (UniqueName: \"kubernetes.io/projected/b1b9a371-da48-4c41-99b3-c77decdd73e6-kube-api-access-48lt5\") pod \"kserve-controller-manager-d6f4b65b8-pgn22\" (UID: \"b1b9a371-da48-4c41-99b3-c77decdd73e6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.557923 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.557904 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w6d\" (UniqueName: \"kubernetes.io/projected/10660b83-85b5-4291-8cd5-ef588ed0898a-kube-api-access-z6w6d\") pod \"llmisvc-controller-manager-777c6bd7d5-tgclp\" (UID: \"10660b83-85b5-4291-8cd5-ef588ed0898a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.665996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.665893 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:31.676222 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.676197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:31.907084 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:30:31.907055 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10660b83_85b5_4291_8cd5_ef588ed0898a.slice/crio-a675525be5f3d287197ea9814242d70c129cef72071db1d703caceb8439b6847 WatchSource:0}: Error finding container a675525be5f3d287197ea9814242d70c129cef72071db1d703caceb8439b6847: Status 404 returned error can't find the container with id a675525be5f3d287197ea9814242d70c129cef72071db1d703caceb8439b6847 Jul 06 20:30:31.912233 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.912206 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pgn22"] Jul 06 20:30:31.915286 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:31.915259 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp"] Jul 06 20:30:32.367874 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:32.367839 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" event={"ID":"b1b9a371-da48-4c41-99b3-c77decdd73e6","Type":"ContainerStarted","Data":"850d42bc9c68152549b5bdd84ec681a5ae9d49289c4823f3d6bb404c08157456"} Jul 06 20:30:32.369987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:32.369960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" event={"ID":"10660b83-85b5-4291-8cd5-ef588ed0898a","Type":"ContainerStarted","Data":"a675525be5f3d287197ea9814242d70c129cef72071db1d703caceb8439b6847"} Jul 06 20:30:32.559573 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:32.559539 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede4e587-d0ca-4de4-8218-e62a5951d87f" path="/var/lib/kubelet/pods/ede4e587-d0ca-4de4-8218-e62a5951d87f/volumes" Jul 06 20:30:36.328323 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.328290 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:30:36.331028 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.331011 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vwcmz" Jul 06 20:30:36.396245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.396203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" event={"ID":"b1b9a371-da48-4c41-99b3-c77decdd73e6","Type":"ContainerStarted","Data":"49630aa752878101a57b0dc11a2bcd288f707769beac702c247cc417aa222902"} Jul 06 20:30:36.396647 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.396545 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:30:36.398656 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.398616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" event={"ID":"10660b83-85b5-4291-8cd5-ef588ed0898a","Type":"ContainerStarted","Data":"fc70f90beed1c3a8509619acb3af057990367814107d1477c5752367d68f7dfb"} Jul 06 20:30:36.398768 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.398738 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:30:36.429467 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.429414 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" podStartSLOduration=1.869203287 podStartE2EDuration="5.429397723s" totalImagesPullingTime="3.560194436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:36.42763793 +0000 UTC m=+378.968694366" watchObservedRunningTime="2026-07-06 20:30:36.429397723 +0000 UTC m=+378.970454159" Jul 06 20:30:36.453140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:30:36.452208 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" podStartSLOduration=1.885002378 podStartE2EDuration="5.452187728s" totalImagesPullingTime="3.56718535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:30:36.450566314 +0000 UTC m=+378.991622750" watchObservedRunningTime="2026-07-06 20:30:36.452187728 +0000 UTC m=+378.993244163" Jul 06 20:31:07.406570 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:07.406535 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tgclp" Jul 06 20:31:07.409724 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:07.409702 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:31:14.379051 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.379017 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:14.382810 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.382788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:14.386806 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.386775 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 20:31:14.386961 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.386852 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-chwcr\"" Jul 06 20:31:14.386961 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.386898 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 20:31:14.394082 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.394059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:14.449954 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.449910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkln\" (UniqueName: \"kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln\") pod \"kuadrant-operator-catalog-bf5pf\" (UID: \"fcc5948a-6e3b-4a3c-a07b-2077f32b1852\") " pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:14.551303 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.551272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkln\" (UniqueName: \"kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln\") pod \"kuadrant-operator-catalog-bf5pf\" (UID: \"fcc5948a-6e3b-4a3c-a07b-2077f32b1852\") " pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:14.559886 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.559855 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkln\" (UniqueName: \"kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln\") pod \"kuadrant-operator-catalog-bf5pf\" (UID: \"fcc5948a-6e3b-4a3c-a07b-2077f32b1852\") " pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:14.694844 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.694755 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:14.744407 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.744033 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:14.828799 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.828770 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:14.831043 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:14.831015 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc5948a_6e3b_4a3c_a07b_2077f32b1852.slice/crio-0f97b0ca62e40902dbb214c2e2584b5986e6bdef7e79031ae99146371e87f93d WatchSource:0}: Error finding container 0f97b0ca62e40902dbb214c2e2584b5986e6bdef7e79031ae99146371e87f93d: Status 404 returned error can't find the container with id 0f97b0ca62e40902dbb214c2e2584b5986e6bdef7e79031ae99146371e87f93d Jul 06 20:31:14.954762 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.954666 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tjz7t"] Jul 06 20:31:14.957795 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.957771 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:14.967175 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:14.967138 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tjz7t"] Jul 06 20:31:15.055425 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.055385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ws4\" (UniqueName: \"kubernetes.io/projected/91892e9d-38a8-42ff-a919-dfdb29cee95f-kube-api-access-79ws4\") pod \"kuadrant-operator-catalog-tjz7t\" (UID: \"91892e9d-38a8-42ff-a919-dfdb29cee95f\") " pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:15.156651 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.156603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ws4\" (UniqueName: \"kubernetes.io/projected/91892e9d-38a8-42ff-a919-dfdb29cee95f-kube-api-access-79ws4\") pod \"kuadrant-operator-catalog-tjz7t\" (UID: \"91892e9d-38a8-42ff-a919-dfdb29cee95f\") " pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:15.165264 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.165234 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ws4\" (UniqueName: \"kubernetes.io/projected/91892e9d-38a8-42ff-a919-dfdb29cee95f-kube-api-access-79ws4\") pod \"kuadrant-operator-catalog-tjz7t\" (UID: \"91892e9d-38a8-42ff-a919-dfdb29cee95f\") " pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:15.269383 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.269281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:15.419994 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.419927 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tjz7t"] Jul 06 20:31:15.456332 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:15.456293 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91892e9d_38a8_42ff_a919_dfdb29cee95f.slice/crio-c8a890bce14d1de16655ada4626b906163cc31b90457c4d0ac71250b40463719 WatchSource:0}: Error finding container c8a890bce14d1de16655ada4626b906163cc31b90457c4d0ac71250b40463719: Status 404 returned error can't find the container with id c8a890bce14d1de16655ada4626b906163cc31b90457c4d0ac71250b40463719 Jul 06 20:31:15.593784 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.593731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" event={"ID":"91892e9d-38a8-42ff-a919-dfdb29cee95f","Type":"ContainerStarted","Data":"c8a890bce14d1de16655ada4626b906163cc31b90457c4d0ac71250b40463719"} Jul 06 20:31:15.594909 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:15.594877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" event={"ID":"fcc5948a-6e3b-4a3c-a07b-2077f32b1852","Type":"ContainerStarted","Data":"0f97b0ca62e40902dbb214c2e2584b5986e6bdef7e79031ae99146371e87f93d"} Jul 06 20:31:17.608220 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.608169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" event={"ID":"91892e9d-38a8-42ff-a919-dfdb29cee95f","Type":"ContainerStarted","Data":"812b15847ffcc55d4d40d1809fd1018d71926ab75bb6d8855d477c3b09582d22"} Jul 06 20:31:17.609586 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.609559 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" event={"ID":"fcc5948a-6e3b-4a3c-a07b-2077f32b1852","Type":"ContainerStarted","Data":"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023"} Jul 06 20:31:17.609708 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.609649 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" podUID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" containerName="registry-server" containerID="cri-o://9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023" gracePeriod=2 Jul 06 20:31:17.625750 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.625706 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" podStartSLOduration=1.971639187 podStartE2EDuration="3.625691746s" totalImagesPullingTime="1.654052559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:17.624688509 +0000 UTC m=+420.165744950" watchObservedRunningTime="2026-07-06 20:31:17.625691746 +0000 UTC m=+420.166748160" Jul 06 20:31:17.644398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.644343 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" podStartSLOduration=1.3674894850000001 podStartE2EDuration="3.644326595s" totalImagesPullingTime="2.27683711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:17.642673401 +0000 UTC m=+420.183729839" watchObservedRunningTime="2026-07-06 20:31:17.644326595 +0000 UTC m=+420.185383031" Jul 06 20:31:17.865898 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.865833 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:17.983946 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.983909 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jkln\" (UniqueName: \"kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln\") pod \"fcc5948a-6e3b-4a3c-a07b-2077f32b1852\" (UID: \"fcc5948a-6e3b-4a3c-a07b-2077f32b1852\") " Jul 06 20:31:17.986201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:17.986173 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln" (OuterVolumeSpecName: "kube-api-access-9jkln") pod "fcc5948a-6e3b-4a3c-a07b-2077f32b1852" (UID: "fcc5948a-6e3b-4a3c-a07b-2077f32b1852"). InnerVolumeSpecName "kube-api-access-9jkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:18.084554 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.084520 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkln\" (UniqueName: \"kubernetes.io/projected/fcc5948a-6e3b-4a3c-a07b-2077f32b1852-kube-api-access-9jkln\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:18.615892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.615856 2562 generic.go:356] "Generic (PLEG): container finished" podID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" containerID="9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023" exitCode=0 Jul 06 20:31:18.616324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.615900 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" Jul 06 20:31:18.616324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.615940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" event={"ID":"fcc5948a-6e3b-4a3c-a07b-2077f32b1852","Type":"ContainerDied","Data":"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023"} Jul 06 20:31:18.616324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.615992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bf5pf" event={"ID":"fcc5948a-6e3b-4a3c-a07b-2077f32b1852","Type":"ContainerDied","Data":"0f97b0ca62e40902dbb214c2e2584b5986e6bdef7e79031ae99146371e87f93d"} Jul 06 20:31:18.616324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.616019 2562 scope.go:117] "RemoveContainer" containerID="9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023" Jul 06 20:31:18.625304 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.625286 2562 scope.go:117] "RemoveContainer" containerID="9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023" Jul 06 20:31:18.625576 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:31:18.625558 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023\": container with ID starting with 9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023 not found: ID does not exist" containerID="9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023" Jul 06 20:31:18.625628 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.625588 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023"} err="failed to get container status \"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023\": rpc error: code = NotFound desc = could not find container \"9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023\": container with ID starting with 9ee43f1b2083a9bdd8f8b5fe8b100f4b26af9df89319ed6cfbdab2c55374e023 not found: ID does not exist" Jul 06 20:31:18.632502 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.632476 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:18.639529 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:18.639505 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bf5pf"] Jul 06 20:31:20.559679 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:20.559643 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" path="/var/lib/kubelet/pods/fcc5948a-6e3b-4a3c-a07b-2077f32b1852/volumes" Jul 06 20:31:25.270241 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:25.270185 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:25.270682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:25.270267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:25.293579 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:25.293545 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:25.674758 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:25.674731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tjz7t" Jul 06 20:31:29.994897 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:29.994860 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt"] Jul 06 20:31:29.995556 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:29.995469 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" containerName="registry-server" Jul 06 20:31:29.995556 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:29.995495 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" containerName="registry-server" Jul 06 20:31:29.995680 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:29.995627 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc5948a-6e3b-4a3c-a07b-2077f32b1852" containerName="registry-server" Jul 06 20:31:30.000484 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.000459 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.002988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.002961 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c4plq\"" Jul 06 20:31:30.007681 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.007649 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt"] Jul 06 20:31:30.097664 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.097626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfklf\" (UniqueName: \"kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.097845 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.097681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.097845 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.097780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.198623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.198582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.198846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.198648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.198846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.198707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfklf\" (UniqueName: \"kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.199086 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.199059 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.199086 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.199078 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.207383 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.207360 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfklf\" (UniqueName: \"kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.311762 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.311730 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:30.447764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.447737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt"] Jul 06 20:31:30.449511 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:30.449483 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode129a427_1610_4c09_9958_bc18174d039b.slice/crio-35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57 WatchSource:0}: Error finding container 35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57: Status 404 returned error can't find the container with id 35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57 Jul 06 20:31:30.677766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.677671 2562 generic.go:356] "Generic (PLEG): container finished" podID="e129a427-1610-4c09-9958-bc18174d039b" containerID="a3d8bbeeebfcc1e9915e519534963619805e9cf629eee3711299f4ecae27e6fc" exitCode=0 Jul 06 20:31:30.677766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.677717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" event={"ID":"e129a427-1610-4c09-9958-bc18174d039b","Type":"ContainerDied","Data":"a3d8bbeeebfcc1e9915e519534963619805e9cf629eee3711299f4ecae27e6fc"} Jul 06 20:31:30.678000 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.677787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" event={"ID":"e129a427-1610-4c09-9958-bc18174d039b","Type":"ContainerStarted","Data":"35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57"} Jul 06 20:31:30.791997 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.791961 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt"] Jul 06 20:31:30.799240 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.796307 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:30.804175 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.804132 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt"] Jul 06 20:31:30.904065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.904033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:30.904065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.904068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2v2w\" (UniqueName: \"kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:30.904312 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:30.904241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.005600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.005503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.005600 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.005557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.006013 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.005672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2v2w\" (UniqueName: \"kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.006013 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.005939 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.006013 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.005937 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.013946 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.013919 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2v2w\" (UniqueName: \"kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.109969 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.109934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:31.189741 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.189703 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt"] Jul 06 20:31:31.193604 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.193581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.204118 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.204086 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt"] Jul 06 20:31:31.255630 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.255576 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt"] Jul 06 20:31:31.257495 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:31.257468 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7f962e_e91f_4b5b_85fd_c8f9584891e6.slice/crio-9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d WatchSource:0}: Error finding container 9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d: Status 404 returned error can't find the container with id 9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d Jul 06 20:31:31.308088 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.308054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.308257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.308125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.308257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.308178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsc9\" (UniqueName: \"kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.409633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.409600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.409801 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.409641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsc9\" (UniqueName: \"kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.409801 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.409691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.410097 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.410081 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.410139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.410091 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.418858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.418829 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsc9\" (UniqueName: \"kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.506749 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.506693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:31.590740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.590633 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl"] Jul 06 20:31:31.594563 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.594540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.603545 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.603523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl"] Jul 06 20:31:31.661284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.661258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt"] Jul 06 20:31:31.663185 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:31.663131 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89982a5e_57bc_49be_a129_c58a1f1033cc.slice/crio-48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec WatchSource:0}: Error finding container 48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec: Status 404 returned error can't find the container with id 48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec Jul 06 20:31:31.684620 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.684593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" event={"ID":"89982a5e-57bc-49be-a129-c58a1f1033cc","Type":"ContainerStarted","Data":"48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec"} Jul 06 20:31:31.685908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.685888 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerID="e8aa8766c29b711d903ffd2f3728193e2f126462e74344cb5d1d5f3066bf3803" exitCode=0 Jul 06 20:31:31.686002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.685971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" event={"ID":"9a7f962e-e91f-4b5b-85fd-c8f9584891e6","Type":"ContainerDied","Data":"e8aa8766c29b711d903ffd2f3728193e2f126462e74344cb5d1d5f3066bf3803"} Jul 06 20:31:31.686077 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.686021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" event={"ID":"9a7f962e-e91f-4b5b-85fd-c8f9584891e6","Type":"ContainerStarted","Data":"9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d"} Jul 06 20:31:31.687695 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.687669 2562 generic.go:356] "Generic (PLEG): container finished" podID="e129a427-1610-4c09-9958-bc18174d039b" containerID="c0967c4a01408e1e9b885202030ba779ef0bbf1b937806b36c30bb4dbd15297b" exitCode=0 Jul 06 20:31:31.687766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.687713 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" event={"ID":"e129a427-1610-4c09-9958-bc18174d039b","Type":"ContainerDied","Data":"c0967c4a01408e1e9b885202030ba779ef0bbf1b937806b36c30bb4dbd15297b"} Jul 06 20:31:31.711528 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.711501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9cn\" (UniqueName: \"kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.711634 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.711612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.711770 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.711685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.812192 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.812131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.812356 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.812218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.812356 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.812253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9cn\" (UniqueName: \"kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.812494 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.812471 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.812588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.812566 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.820757 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.820736 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9cn\" (UniqueName: \"kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:31.949721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:31.949682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:32.087403 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.087373 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl"] Jul 06 20:31:32.088852 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:32.088827 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe2c00d_1094_48cf_93b6_1118d47c319f.slice/crio-97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8 WatchSource:0}: Error finding container 97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8: Status 404 returned error can't find the container with id 97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8 Jul 06 20:31:32.695141 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.695049 2562 generic.go:356] "Generic (PLEG): container finished" podID="e129a427-1610-4c09-9958-bc18174d039b" containerID="40f21bacef77f180c5b9658ecaef2a37de36005507b73bef2e9fa0ae0fd1c4f3" exitCode=0 Jul 06 20:31:32.695326 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.695143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" event={"ID":"e129a427-1610-4c09-9958-bc18174d039b","Type":"ContainerDied","Data":"40f21bacef77f180c5b9658ecaef2a37de36005507b73bef2e9fa0ae0fd1c4f3"} Jul 06 20:31:32.696508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.696483 2562 generic.go:356] "Generic (PLEG): container finished" podID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerID="ec2fa00229d0701fa7abde8be9c5af43edd322eabb3b96b6259226ca042a1fe4" exitCode=0 Jul 06 20:31:32.696615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.696567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" event={"ID":"0fe2c00d-1094-48cf-93b6-1118d47c319f","Type":"ContainerDied","Data":"ec2fa00229d0701fa7abde8be9c5af43edd322eabb3b96b6259226ca042a1fe4"} Jul 06 20:31:32.696615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.696610 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" event={"ID":"0fe2c00d-1094-48cf-93b6-1118d47c319f","Type":"ContainerStarted","Data":"97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8"} Jul 06 20:31:32.698268 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.698243 2562 generic.go:356] "Generic (PLEG): container finished" podID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerID="f809ad0234b21dbca4f3ed0f446b9a85f02d741a04c8c17e36238fb37ac6d2c5" exitCode=0 Jul 06 20:31:32.698347 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.698320 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" event={"ID":"89982a5e-57bc-49be-a129-c58a1f1033cc","Type":"ContainerDied","Data":"f809ad0234b21dbca4f3ed0f446b9a85f02d741a04c8c17e36238fb37ac6d2c5"} Jul 06 20:31:32.700385 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.700314 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerID="07433b8da49f5c8f134a52590929418d9a95ab5805598aff764b17eda2ae4a35" exitCode=0 Jul 06 20:31:32.700385 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:32.700349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" event={"ID":"9a7f962e-e91f-4b5b-85fd-c8f9584891e6","Type":"ContainerDied","Data":"07433b8da49f5c8f134a52590929418d9a95ab5805598aff764b17eda2ae4a35"} Jul 06 20:31:33.713034 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.712995 2562 generic.go:356] "Generic (PLEG): container finished" podID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerID="d76886465f33c5566c6b6ea361665099bf74eb7b8746c9125011f3ab8e40d56c" exitCode=0 Jul 06 20:31:33.713508 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.713074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" event={"ID":"89982a5e-57bc-49be-a129-c58a1f1033cc","Type":"ContainerDied","Data":"d76886465f33c5566c6b6ea361665099bf74eb7b8746c9125011f3ab8e40d56c"} Jul 06 20:31:33.715396 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.715375 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerID="d8cc80f79f91808915f280b047140ee860c02c75f15b81893852a6c7b94e44dd" exitCode=0 Jul 06 20:31:33.715516 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.715452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" event={"ID":"9a7f962e-e91f-4b5b-85fd-c8f9584891e6","Type":"ContainerDied","Data":"d8cc80f79f91808915f280b047140ee860c02c75f15b81893852a6c7b94e44dd"} Jul 06 20:31:33.717184 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.717163 2562 generic.go:356] "Generic (PLEG): container finished" podID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerID="e3bc9b6ce24b809be07faaf80c347a45424ced5845fc67118bebd1bef2ee30a3" exitCode=0 Jul 06 20:31:33.717275 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.717192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" event={"ID":"0fe2c00d-1094-48cf-93b6-1118d47c319f","Type":"ContainerDied","Data":"e3bc9b6ce24b809be07faaf80c347a45424ced5845fc67118bebd1bef2ee30a3"} Jul 06 20:31:33.850166 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.850122 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:33.931167 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.931122 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfklf\" (UniqueName: \"kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf\") pod \"e129a427-1610-4c09-9958-bc18174d039b\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " Jul 06 20:31:33.931310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.931192 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle\") pod \"e129a427-1610-4c09-9958-bc18174d039b\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " Jul 06 20:31:33.931310 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.931253 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util\") pod \"e129a427-1610-4c09-9958-bc18174d039b\" (UID: \"e129a427-1610-4c09-9958-bc18174d039b\") " Jul 06 20:31:33.931666 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.931637 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle" (OuterVolumeSpecName: "bundle") pod "e129a427-1610-4c09-9958-bc18174d039b" (UID: "e129a427-1610-4c09-9958-bc18174d039b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:33.933067 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.933048 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf" (OuterVolumeSpecName: "kube-api-access-vfklf") pod "e129a427-1610-4c09-9958-bc18174d039b" (UID: "e129a427-1610-4c09-9958-bc18174d039b"). InnerVolumeSpecName "kube-api-access-vfklf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:33.937253 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:33.937222 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util" (OuterVolumeSpecName: "util") pod "e129a427-1610-4c09-9958-bc18174d039b" (UID: "e129a427-1610-4c09-9958-bc18174d039b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:34.032477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.032432 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfklf\" (UniqueName: \"kubernetes.io/projected/e129a427-1610-4c09-9958-bc18174d039b-kube-api-access-vfklf\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:34.032477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.032470 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:34.032477 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.032479 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e129a427-1610-4c09-9958-bc18174d039b-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:34.724248 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.724204 2562 generic.go:356] "Generic (PLEG): container finished" podID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerID="5338d5eb039ddb99cbcf41a0540b3f5aa89261320ba4eaac3dcf486971333c25" exitCode=0 Jul 06 20:31:34.724654 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.724280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" event={"ID":"89982a5e-57bc-49be-a129-c58a1f1033cc","Type":"ContainerDied","Data":"5338d5eb039ddb99cbcf41a0540b3f5aa89261320ba4eaac3dcf486971333c25"} Jul 06 20:31:34.726221 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.726199 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" Jul 06 20:31:34.726352 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.726205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt" event={"ID":"e129a427-1610-4c09-9958-bc18174d039b","Type":"ContainerDied","Data":"35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57"} Jul 06 20:31:34.726352 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.726320 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ea43a5901ec75a20c5a3b350c2412372d6b9a8cd0b18399656e6aca164bb57" Jul 06 20:31:34.728703 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.728677 2562 generic.go:356] "Generic (PLEG): container finished" podID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerID="cea888239db430ab0d30ad9a1d10d9eb99dc4917e5c62dfadf2aea02f04dc421" exitCode=0 Jul 06 20:31:34.728814 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.728792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" event={"ID":"0fe2c00d-1094-48cf-93b6-1118d47c319f","Type":"ContainerDied","Data":"cea888239db430ab0d30ad9a1d10d9eb99dc4917e5c62dfadf2aea02f04dc421"} Jul 06 20:31:34.864640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:34.864607 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:35.041694 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.041662 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2v2w\" (UniqueName: \"kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w\") pod \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " Jul 06 20:31:35.041694 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.041702 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle\") pod \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " Jul 06 20:31:35.041928 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.041738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util\") pod \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\" (UID: \"9a7f962e-e91f-4b5b-85fd-c8f9584891e6\") " Jul 06 20:31:35.042367 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.042339 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle" (OuterVolumeSpecName: "bundle") pod "9a7f962e-e91f-4b5b-85fd-c8f9584891e6" (UID: "9a7f962e-e91f-4b5b-85fd-c8f9584891e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:35.043941 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.043909 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w" (OuterVolumeSpecName: "kube-api-access-x2v2w") pod "9a7f962e-e91f-4b5b-85fd-c8f9584891e6" (UID: "9a7f962e-e91f-4b5b-85fd-c8f9584891e6"). InnerVolumeSpecName "kube-api-access-x2v2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:35.048016 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.047991 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util" (OuterVolumeSpecName: "util") pod "9a7f962e-e91f-4b5b-85fd-c8f9584891e6" (UID: "9a7f962e-e91f-4b5b-85fd-c8f9584891e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:35.142788 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.142695 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2v2w\" (UniqueName: \"kubernetes.io/projected/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-kube-api-access-x2v2w\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:35.142788 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.142725 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:35.142788 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.142736 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7f962e-e91f-4b5b-85fd-c8f9584891e6-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:35.736567 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.736525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" event={"ID":"9a7f962e-e91f-4b5b-85fd-c8f9584891e6","Type":"ContainerDied","Data":"9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d"} Jul 06 20:31:35.736567 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.736567 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt" Jul 06 20:31:35.737092 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.736591 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f16d3b93046a5e6341ae5f492f0a2b02f0c90d139b2884541719d72ef6a9f1d" Jul 06 20:31:35.900990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.900965 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:35.930164 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:35.930124 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:36.050972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.050873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle\") pod \"0fe2c00d-1094-48cf-93b6-1118d47c319f\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " Jul 06 20:31:36.050972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.050918 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle\") pod \"89982a5e-57bc-49be-a129-c58a1f1033cc\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " Jul 06 20:31:36.050972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.050939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwsc9\" (UniqueName: \"kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9\") pod \"89982a5e-57bc-49be-a129-c58a1f1033cc\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " Jul 06 20:31:36.050972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.050964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util\") pod \"0fe2c00d-1094-48cf-93b6-1118d47c319f\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " Jul 06 20:31:36.051351 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.051039 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util\") pod \"89982a5e-57bc-49be-a129-c58a1f1033cc\" (UID: \"89982a5e-57bc-49be-a129-c58a1f1033cc\") " Jul 06 20:31:36.051351 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.051065 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx9cn\" (UniqueName: \"kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn\") pod \"0fe2c00d-1094-48cf-93b6-1118d47c319f\" (UID: \"0fe2c00d-1094-48cf-93b6-1118d47c319f\") " Jul 06 20:31:36.051607 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.051570 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle" (OuterVolumeSpecName: "bundle") pod "0fe2c00d-1094-48cf-93b6-1118d47c319f" (UID: "0fe2c00d-1094-48cf-93b6-1118d47c319f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:36.051607 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.051583 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle" (OuterVolumeSpecName: "bundle") pod "89982a5e-57bc-49be-a129-c58a1f1033cc" (UID: "89982a5e-57bc-49be-a129-c58a1f1033cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:36.053606 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.053581 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9" (OuterVolumeSpecName: "kube-api-access-dwsc9") pod "89982a5e-57bc-49be-a129-c58a1f1033cc" (UID: "89982a5e-57bc-49be-a129-c58a1f1033cc"). InnerVolumeSpecName "kube-api-access-dwsc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:36.053700 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.053606 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn" (OuterVolumeSpecName: "kube-api-access-gx9cn") pod "0fe2c00d-1094-48cf-93b6-1118d47c319f" (UID: "0fe2c00d-1094-48cf-93b6-1118d47c319f"). InnerVolumeSpecName "kube-api-access-gx9cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:36.057214 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.057189 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util" (OuterVolumeSpecName: "util") pod "89982a5e-57bc-49be-a129-c58a1f1033cc" (UID: "89982a5e-57bc-49be-a129-c58a1f1033cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:36.057702 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.057676 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util" (OuterVolumeSpecName: "util") pod "0fe2c00d-1094-48cf-93b6-1118d47c319f" (UID: "0fe2c00d-1094-48cf-93b6-1118d47c319f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:31:36.152872 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152815 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.152872 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152866 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsc9\" (UniqueName: \"kubernetes.io/projected/89982a5e-57bc-49be-a129-c58a1f1033cc-kube-api-access-dwsc9\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.152872 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152881 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.153125 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152893 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89982a5e-57bc-49be-a129-c58a1f1033cc-util\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.153125 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152905 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9cn\" (UniqueName: \"kubernetes.io/projected/0fe2c00d-1094-48cf-93b6-1118d47c319f-kube-api-access-gx9cn\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.153125 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.152917 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe2c00d-1094-48cf-93b6-1118d47c319f-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:31:36.744340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.744303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" event={"ID":"89982a5e-57bc-49be-a129-c58a1f1033cc","Type":"ContainerDied","Data":"48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec"} Jul 06 20:31:36.744340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.744324 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt" Jul 06 20:31:36.744340 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.744356 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b4fdddb2fd16173591b84726dfba7d85b643d650fd3555cb3237919bb634ec" Jul 06 20:31:36.746387 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.746340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" event={"ID":"0fe2c00d-1094-48cf-93b6-1118d47c319f","Type":"ContainerDied","Data":"97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8"} Jul 06 20:31:36.746387 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.746385 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97be3aa55185503c7eaeb920d9748d947537709c32dea860a265ccb97a7d8af8" Jul 06 20:31:36.746595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:36.746352 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl" Jul 06 20:31:48.230321 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230234 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8"] Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230629 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="util" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230640 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="util" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230648 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="pull" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230654 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="pull" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230667 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="pull" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230673 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="pull" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230685 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="pull" Jul 06 20:31:48.230690 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230690 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="pull" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230697 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230702 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230707 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="pull" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230712 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="pull" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230720 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230725 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230734 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230739 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230746 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230750 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230757 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230762 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230767 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230772 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230779 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230784 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="util" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230838 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7f962e-e91f-4b5b-85fd-c8f9584891e6" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230848 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="89982a5e-57bc-49be-a129-c58a1f1033cc" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230862 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe2c00d-1094-48cf-93b6-1118d47c319f" containerName="extract" Jul 06 20:31:48.230944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.230870 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e129a427-1610-4c09-9958-bc18174d039b" containerName="extract" Jul 06 20:31:48.233992 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.233971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.236930 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.236908 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qtc9k\"" Jul 06 20:31:48.251294 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.251262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8"] Jul 06 20:31:48.264415 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.264382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.264577 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.264418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5bj\" (UniqueName: \"kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.365277 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.365234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.365277 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.365277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5bj\" (UniqueName: \"kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.365639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.365618 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.380234 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.380203 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5bj\" (UniqueName: \"kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.544178 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.544072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:48.698429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.698394 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8"] Jul 06 20:31:48.699287 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:48.699260 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64412062_b96e_4c26_8c97_b8370f8b0cdb.slice/crio-a20633f47bf8672dad69757721b1cbeed47e49e6c02dccd765b8d2c9dd44a650 WatchSource:0}: Error finding container a20633f47bf8672dad69757721b1cbeed47e49e6c02dccd765b8d2c9dd44a650: Status 404 returned error can't find the container with id a20633f47bf8672dad69757721b1cbeed47e49e6c02dccd765b8d2c9dd44a650 Jul 06 20:31:48.812726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:48.812696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" event={"ID":"64412062-b96e-4c26-8c97-b8370f8b0cdb","Type":"ContainerStarted","Data":"a20633f47bf8672dad69757721b1cbeed47e49e6c02dccd765b8d2c9dd44a650"} Jul 06 20:31:50.843747 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.843717 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk"] Jul 06 20:31:50.848278 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.848259 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:50.852178 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.852125 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-r57lt\"" Jul 06 20:31:50.867461 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.867428 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk"] Jul 06 20:31:50.890111 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.890079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6p5\" (UniqueName: \"kubernetes.io/projected/deaa7ac1-a677-4ae7-8651-c26c8b12ab15-kube-api-access-7n6p5\") pod \"authorino-operator-75cc8d5cb7-v2tgk\" (UID: \"deaa7ac1-a677-4ae7-8651-c26c8b12ab15\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:50.991171 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:50.991110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6p5\" (UniqueName: \"kubernetes.io/projected/deaa7ac1-a677-4ae7-8651-c26c8b12ab15-kube-api-access-7n6p5\") pod \"authorino-operator-75cc8d5cb7-v2tgk\" (UID: \"deaa7ac1-a677-4ae7-8651-c26c8b12ab15\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:51.002700 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:51.002656 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6p5\" (UniqueName: \"kubernetes.io/projected/deaa7ac1-a677-4ae7-8651-c26c8b12ab15-kube-api-access-7n6p5\") pod \"authorino-operator-75cc8d5cb7-v2tgk\" (UID: \"deaa7ac1-a677-4ae7-8651-c26c8b12ab15\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:51.159279 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:51.159186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:51.329752 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:51.329723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk"] Jul 06 20:31:51.331202 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:51.331139 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa7ac1_a677_4ae7_8651_c26c8b12ab15.slice/crio-5769d4a716c292f22c7e036353e1c9c5d2a627135f5a283d28565e11ee535e28 WatchSource:0}: Error finding container 5769d4a716c292f22c7e036353e1c9c5d2a627135f5a283d28565e11ee535e28: Status 404 returned error can't find the container with id 5769d4a716c292f22c7e036353e1c9c5d2a627135f5a283d28565e11ee535e28 Jul 06 20:31:51.831463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:51.831425 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" event={"ID":"deaa7ac1-a677-4ae7-8651-c26c8b12ab15","Type":"ContainerStarted","Data":"5769d4a716c292f22c7e036353e1c9c5d2a627135f5a283d28565e11ee535e28"} Jul 06 20:31:53.580638 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.580593 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9"] Jul 06 20:31:53.584520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.584498 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:53.588440 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.588419 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cvtf2\"" Jul 06 20:31:53.588742 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.588728 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 20:31:53.605332 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.605300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9"] Jul 06 20:31:53.615529 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.615223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w7s\" (UniqueName: \"kubernetes.io/projected/ab8a2440-71eb-45d3-8399-d42a58abdc15-kube-api-access-92w7s\") pod \"dns-operator-controller-manager-864dd78c4c-l4nb9\" (UID: \"ab8a2440-71eb-45d3-8399-d42a58abdc15\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:53.716088 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.716013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w7s\" (UniqueName: \"kubernetes.io/projected/ab8a2440-71eb-45d3-8399-d42a58abdc15-kube-api-access-92w7s\") pod \"dns-operator-controller-manager-864dd78c4c-l4nb9\" (UID: \"ab8a2440-71eb-45d3-8399-d42a58abdc15\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:53.731647 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.731612 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w7s\" (UniqueName: \"kubernetes.io/projected/ab8a2440-71eb-45d3-8399-d42a58abdc15-kube-api-access-92w7s\") pod \"dns-operator-controller-manager-864dd78c4c-l4nb9\" (UID: \"ab8a2440-71eb-45d3-8399-d42a58abdc15\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:53.897400 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:53.897379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:54.063497 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:54.063463 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9"] Jul 06 20:31:54.065338 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:54.065304 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8a2440_71eb_45d3_8399_d42a58abdc15.slice/crio-2eec77dc7decb942d9d1f23a4e9a5fe2650ceec1f757ae11e740a90e7f5a7de8 WatchSource:0}: Error finding container 2eec77dc7decb942d9d1f23a4e9a5fe2650ceec1f757ae11e740a90e7f5a7de8: Status 404 returned error can't find the container with id 2eec77dc7decb942d9d1f23a4e9a5fe2650ceec1f757ae11e740a90e7f5a7de8 Jul 06 20:31:54.855068 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:54.855027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" event={"ID":"ab8a2440-71eb-45d3-8399-d42a58abdc15","Type":"ContainerStarted","Data":"2eec77dc7decb942d9d1f23a4e9a5fe2650ceec1f757ae11e740a90e7f5a7de8"} Jul 06 20:31:54.856975 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:54.856943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" event={"ID":"deaa7ac1-a677-4ae7-8651-c26c8b12ab15","Type":"ContainerStarted","Data":"d0902cef60e9cec77a91e1cdadef0c1a6a5d683c76eae3c695b40d11161184b0"} Jul 06 20:31:54.857108 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:54.857039 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:31:54.881925 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:54.881864 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" podStartSLOduration=2.322464416 podStartE2EDuration="4.88184741s" totalImagesPullingTime="2.559382994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:54.879689966 +0000 UTC m=+457.420746404" watchObservedRunningTime="2026-07-06 20:31:54.88184741 +0000 UTC m=+457.422903846" Jul 06 20:31:56.870838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:56.870797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" event={"ID":"64412062-b96e-4c26-8c97-b8370f8b0cdb","Type":"ContainerStarted","Data":"f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b"} Jul 06 20:31:56.871348 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:56.870875 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:31:56.901817 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:56.901754 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" podStartSLOduration=1.73239472 podStartE2EDuration="8.901726587s" totalImagesPullingTime="7.169331867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:56.897317958 +0000 UTC m=+459.438374420" watchObservedRunningTime="2026-07-06 20:31:56.901726587 +0000 UTC m=+459.442783097" Jul 06 20:31:58.845806 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.845766 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t"] Jul 06 20:31:58.849364 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.849347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:58.851950 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.851926 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 20:31:58.852075 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.851985 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c4plq\"" Jul 06 20:31:58.852075 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.852065 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 20:31:58.858170 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.858130 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t"] Jul 06 20:31:58.882794 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.882758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" event={"ID":"ab8a2440-71eb-45d3-8399-d42a58abdc15","Type":"ContainerStarted","Data":"7adb0424445f3ff91406f724383b51321f42d604d99ab7db59f01675fff9c778"} Jul 06 20:31:58.882970 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.882840 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:31:58.905074 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.905022 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" podStartSLOduration=2.13615872 podStartE2EDuration="5.90500635s" totalImagesPullingTime="3.76884763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:58.90375226 +0000 UTC m=+461.444808701" watchObservedRunningTime="2026-07-06 20:31:58.90500635 +0000 UTC m=+461.446062826" Jul 06 20:31:58.963312 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.963270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bad6ecf-539b-4f93-826e-1e45aa557187-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:58.963312 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.963311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bad6ecf-539b-4f93-826e-1e45aa557187-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:58.963537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:58.963337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx95\" (UniqueName: \"kubernetes.io/projected/8bad6ecf-539b-4f93-826e-1e45aa557187-kube-api-access-2tx95\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.064069 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.064035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bad6ecf-539b-4f93-826e-1e45aa557187-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.064069 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.064074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bad6ecf-539b-4f93-826e-1e45aa557187-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.064316 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.064098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx95\" (UniqueName: \"kubernetes.io/projected/8bad6ecf-539b-4f93-826e-1e45aa557187-kube-api-access-2tx95\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.064785 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.064753 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bad6ecf-539b-4f93-826e-1e45aa557187-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.066618 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.066599 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bad6ecf-539b-4f93-826e-1e45aa557187-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.072910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.072879 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx95\" (UniqueName: \"kubernetes.io/projected/8bad6ecf-539b-4f93-826e-1e45aa557187-kube-api-access-2tx95\") pod \"kuadrant-console-plugin-66b5774985-znt5t\" (UID: \"8bad6ecf-539b-4f93-826e-1e45aa557187\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.159776 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.159673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" Jul 06 20:31:59.305402 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.305375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t"] Jul 06 20:31:59.307168 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:31:59.307124 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bad6ecf_539b_4f93_826e_1e45aa557187.slice/crio-1882b8e16dc1c07ac392d0d2d10e54ff672531fd8d4aafc3d634630ff78d979a WatchSource:0}: Error finding container 1882b8e16dc1c07ac392d0d2d10e54ff672531fd8d4aafc3d634630ff78d979a: Status 404 returned error can't find the container with id 1882b8e16dc1c07ac392d0d2d10e54ff672531fd8d4aafc3d634630ff78d979a Jul 06 20:31:59.890354 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:31:59.890316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" event={"ID":"8bad6ecf-539b-4f93-826e-1e45aa557187","Type":"ContainerStarted","Data":"1882b8e16dc1c07ac392d0d2d10e54ff672531fd8d4aafc3d634630ff78d979a"} Jul 06 20:32:05.865928 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:05.865894 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2tgk" Jul 06 20:32:07.878180 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:07.878125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:09.371933 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.371888 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b7599c8b-m427z"] Jul 06 20:32:09.382331 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.382294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.391546 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.391457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b7599c8b-m427z"] Jul 06 20:32:09.469087 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469293 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-trusted-ca-bundle\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469293 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-oauth-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469293 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4sv\" (UniqueName: \"kubernetes.io/projected/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-kube-api-access-cr4sv\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-oauth-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-service-ca\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.469463 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.469444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.570909 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.570871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-service-ca\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.570909 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.570910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.570944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.570969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-trusted-ca-bundle\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.571034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-oauth-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.571056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4sv\" (UniqueName: \"kubernetes.io/projected/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-kube-api-access-cr4sv\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.571104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-oauth-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.571803 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.571753 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.572123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.572095 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-oauth-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.572504 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.572482 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-service-ca\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.572825 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.572793 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-trusted-ca-bundle\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.573990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.573967 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-oauth-config\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.573990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.573982 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-console-serving-cert\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.585329 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.585307 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4sv\" (UniqueName: \"kubernetes.io/projected/f99b3e69-6565-44c2-bf60-0bc684b9ab5b-kube-api-access-cr4sv\") pod \"console-55b7599c8b-m427z\" (UID: \"f99b3e69-6565-44c2-bf60-0bc684b9ab5b\") " pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.699007 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.698908 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:09.705419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.705391 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8"] Jul 06 20:32:09.705632 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.705580 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" containerName="manager" containerID="cri-o://f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b" gracePeriod=2 Jul 06 20:32:09.717067 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.716408 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8"] Jul 06 20:32:09.731188 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.731141 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:09.731635 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.731617 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" containerName="manager" Jul 06 20:32:09.731781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.731638 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" containerName="manager" Jul 06 20:32:09.731781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.731770 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" containerName="manager" Jul 06 20:32:09.735580 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.735559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.739052 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:09.739017 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:09.751164 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.751116 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:09.874120 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.874073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.874321 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.874206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dht7\" (UniqueName: \"kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.892549 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.892518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-l4nb9" Jul 06 20:32:09.925842 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:09.925791 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:09.975139 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.975048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.975307 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.975171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dht7\" (UniqueName: \"kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.975526 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.975499 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:09.988065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:09.988035 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dht7\" (UniqueName: \"kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v2d5c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:10.080257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.080219 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:10.682098 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.682058 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:32:10.690668 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.690642 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.704737 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.704709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:32:10.783371 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.783322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzb4\" (UniqueName: \"kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.783566 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.783506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.884196 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.884130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.884396 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.884233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzb4\" (UniqueName: \"kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.884542 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.884516 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:10.894279 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:10.894247 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzb4\" (UniqueName: \"kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4\") pod \"kuadrant-operator-controller-manager-86878c769f-cqstn\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:11.004913 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:11.004826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:22.932385 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:22.932356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:22.936227 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:22.936188 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:22.951838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:22.951327 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b7599c8b-m427z"] Jul 06 20:32:22.952685 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:32:22.952654 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99b3e69_6565_44c2_bf60_0bc684b9ab5b.slice/crio-d9080650c439e90df40ff1a8eef98e29b85195962bdf54b758d402ac46e7f141 WatchSource:0}: Error finding container d9080650c439e90df40ff1a8eef98e29b85195962bdf54b758d402ac46e7f141: Status 404 returned error can't find the container with id d9080650c439e90df40ff1a8eef98e29b85195962bdf54b758d402ac46e7f141 Jul 06 20:32:23.046045 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.046005 2562 generic.go:356] "Generic (PLEG): container finished" podID="64412062-b96e-4c26-8c97-b8370f8b0cdb" containerID="f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b" exitCode=0 Jul 06 20:32:23.046268 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.046071 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:23.046268 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.046101 2562 scope.go:117] "RemoveContainer" containerID="f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b" Jul 06 20:32:23.047933 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.047888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b7599c8b-m427z" event={"ID":"f99b3e69-6565-44c2-bf60-0bc684b9ab5b","Type":"ContainerStarted","Data":"8d72eed6b683b0fb80d24e0ea66fec38d7a606849788054f28a3dcd05f249c76"} Jul 06 20:32:23.048038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.047994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b7599c8b-m427z" event={"ID":"f99b3e69-6565-44c2-bf60-0bc684b9ab5b","Type":"ContainerStarted","Data":"d9080650c439e90df40ff1a8eef98e29b85195962bdf54b758d402ac46e7f141"} Jul 06 20:32:23.048396 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:23.048362 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:23.049641 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.049619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" event={"ID":"8bad6ecf-539b-4f93-826e-1e45aa557187","Type":"ContainerStarted","Data":"852c2438f831197fe933fb6da818b43faf6cf336b3e42b797db20c575fd939fc"} Jul 06 20:32:23.051697 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:23.051672 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:23.056018 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.055998 2562 scope.go:117] "RemoveContainer" containerID="f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b" Jul 06 20:32:23.056324 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:23.056299 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b\": container with ID starting with f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b not found: ID does not exist" containerID="f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b" Jul 06 20:32:23.056393 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.056336 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b"} err="failed to get container status \"f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b\": rpc error: code = NotFound desc = could not find container \"f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b\": container with ID starting with f9f5bc11724cb2a2ca6690439f325c2d7ff0b362ba07cbb75e89b8d55a1aad3b not found: ID does not exist" Jul 06 20:32:23.082391 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.082330 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b7599c8b-m427z" podStartSLOduration=14.082313084 podStartE2EDuration="14.082313084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:23.07992444 +0000 UTC m=+485.620980889" watchObservedRunningTime="2026-07-06 20:32:23.082313084 +0000 UTC m=+485.623369589" Jul 06 20:32:23.103227 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.103197 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume\") pod \"64412062-b96e-4c26-8c97-b8370f8b0cdb\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " Jul 06 20:32:23.103404 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.103371 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x5bj\" (UniqueName: \"kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj\") pod \"64412062-b96e-4c26-8c97-b8370f8b0cdb\" (UID: \"64412062-b96e-4c26-8c97-b8370f8b0cdb\") " Jul 06 20:32:23.103777 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.103749 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "64412062-b96e-4c26-8c97-b8370f8b0cdb" (UID: "64412062-b96e-4c26-8c97-b8370f8b0cdb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:32:23.105580 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.105549 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj" (OuterVolumeSpecName: "kube-api-access-9x5bj") pod "64412062-b96e-4c26-8c97-b8370f8b0cdb" (UID: "64412062-b96e-4c26-8c97-b8370f8b0cdb"). InnerVolumeSpecName "kube-api-access-9x5bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:23.106184 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:23.106131 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:23.106583 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.106545 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-znt5t" podStartSLOduration=1.5392242139999999 podStartE2EDuration="25.106528826s" totalImagesPullingTime="23.567304612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:23.102861039 +0000 UTC m=+485.643917478" watchObservedRunningTime="2026-07-06 20:32:23.106528826 +0000 UTC m=+485.647585263" Jul 06 20:32:23.200613 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.200588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:32:23.203127 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:32:23.203092 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1b357_e4c9_4366_a72f_cc3751672124.slice/crio-1c68d12e5723e620a31fbe568330e31a8e5c522569c6e7b00055d0c59fd45f50 WatchSource:0}: Error finding container 1c68d12e5723e620a31fbe568330e31a8e5c522569c6e7b00055d0c59fd45f50: Status 404 returned error can't find the container with id 1c68d12e5723e620a31fbe568330e31a8e5c522569c6e7b00055d0c59fd45f50 Jul 06 20:32:23.203618 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:32:23.203591 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca023a_1c80_4812_bb0e_4dc4a871932c.slice/crio-72a35af99dd07f9e4f7c896b701778126a89e4b820c9947e62c0ebf74cc26342 WatchSource:0}: Error finding container 72a35af99dd07f9e4f7c896b701778126a89e4b820c9947e62c0ebf74cc26342: Status 404 returned error can't find the container with id 72a35af99dd07f9e4f7c896b701778126a89e4b820c9947e62c0ebf74cc26342 Jul 06 20:32:23.203776 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.203755 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:23.204740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.204717 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5bj\" (UniqueName: \"kubernetes.io/projected/64412062-b96e-4c26-8c97-b8370f8b0cdb-kube-api-access-9x5bj\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:23.204830 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:23.204747 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64412062-b96e-4c26-8c97-b8370f8b0cdb-extensions-socket-volume\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:23.358542 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:23.358510 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:24.057537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.057426 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" event={"ID":"22f1b357-e4c9-4366-a72f-cc3751672124","Type":"ContainerStarted","Data":"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee"} Jul 06 20:32:24.057537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.057490 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:24.057537 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.057509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" event={"ID":"22f1b357-e4c9-4366-a72f-cc3751672124","Type":"ContainerStarted","Data":"1c68d12e5723e620a31fbe568330e31a8e5c522569c6e7b00055d0c59fd45f50"} Jul 06 20:32:24.059208 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.059177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" event={"ID":"52ca023a-1c80-4812-bb0e-4dc4a871932c","Type":"ContainerStarted","Data":"12d1f55f20b117d559cc13e02d487ce35435ed6b0ccc37be0532918aaf2f6cc6"} Jul 06 20:32:24.059365 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.059232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" event={"ID":"52ca023a-1c80-4812-bb0e-4dc4a871932c","Type":"ContainerStarted","Data":"72a35af99dd07f9e4f7c896b701778126a89e4b820c9947e62c0ebf74cc26342"} Jul 06 20:32:24.059365 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.059261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:24.090328 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.090256 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" podStartSLOduration=14.090238507 podStartE2EDuration="14.090238507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:24.088958051 +0000 UTC m=+486.630014489" watchObservedRunningTime="2026-07-06 20:32:24.090238507 +0000 UTC m=+486.631294944" Jul 06 20:32:24.091202 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:24.091164 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:24.093106 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:24.093086 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qvfw8\" is forbidden: User \"system:node:ip-10-0-134-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-98.ec2.internal' and this object" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qvfw8" Jul 06 20:32:24.130642 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.130584 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" podStartSLOduration=15.130568727 podStartE2EDuration="15.130568727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:24.129653671 +0000 UTC m=+486.670710109" watchObservedRunningTime="2026-07-06 20:32:24.130568727 +0000 UTC m=+486.671625164" Jul 06 20:32:24.560458 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:24.560423 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64412062-b96e-4c26-8c97-b8370f8b0cdb" path="/var/lib/kubelet/pods/64412062-b96e-4c26-8c97-b8370f8b0cdb/volumes" Jul 06 20:32:29.699816 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:29.699778 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:29.700298 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:29.699851 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:29.704660 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:29.704636 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:30.101595 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:30.101564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55b7599c8b-m427z" Jul 06 20:32:30.173597 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:30.173554 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:32:35.069498 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:35.069467 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:32:35.070091 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:35.069549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:35.153247 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:35.153204 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:35.153456 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:35.153425 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" podUID="52ca023a-1c80-4812-bb0e-4dc4a871932c" containerName="manager" containerID="cri-o://12d1f55f20b117d559cc13e02d487ce35435ed6b0ccc37be0532918aaf2f6cc6" gracePeriod=10 Jul 06 20:32:36.134383 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.134344 2562 generic.go:356] "Generic (PLEG): container finished" podID="52ca023a-1c80-4812-bb0e-4dc4a871932c" containerID="12d1f55f20b117d559cc13e02d487ce35435ed6b0ccc37be0532918aaf2f6cc6" exitCode=0 Jul 06 20:32:36.134796 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.134413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" event={"ID":"52ca023a-1c80-4812-bb0e-4dc4a871932c","Type":"ContainerDied","Data":"12d1f55f20b117d559cc13e02d487ce35435ed6b0ccc37be0532918aaf2f6cc6"} Jul 06 20:32:36.560507 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.560479 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:36.645734 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.645701 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dht7\" (UniqueName: \"kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7\") pod \"52ca023a-1c80-4812-bb0e-4dc4a871932c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " Jul 06 20:32:36.645934 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.645755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume\") pod \"52ca023a-1c80-4812-bb0e-4dc4a871932c\" (UID: \"52ca023a-1c80-4812-bb0e-4dc4a871932c\") " Jul 06 20:32:36.646121 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.646091 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "52ca023a-1c80-4812-bb0e-4dc4a871932c" (UID: "52ca023a-1c80-4812-bb0e-4dc4a871932c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:32:36.647973 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.647949 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7" (OuterVolumeSpecName: "kube-api-access-4dht7") pod "52ca023a-1c80-4812-bb0e-4dc4a871932c" (UID: "52ca023a-1c80-4812-bb0e-4dc4a871932c"). InnerVolumeSpecName "kube-api-access-4dht7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:36.746892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.746802 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dht7\" (UniqueName: \"kubernetes.io/projected/52ca023a-1c80-4812-bb0e-4dc4a871932c-kube-api-access-4dht7\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:36.746892 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:36.746837 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52ca023a-1c80-4812-bb0e-4dc4a871932c-extensions-socket-volume\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:37.147409 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:37.147379 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" Jul 06 20:32:37.147409 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:37.147394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c" event={"ID":"52ca023a-1c80-4812-bb0e-4dc4a871932c","Type":"ContainerDied","Data":"72a35af99dd07f9e4f7c896b701778126a89e4b820c9947e62c0ebf74cc26342"} Jul 06 20:32:37.147937 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:37.147479 2562 scope.go:117] "RemoveContainer" containerID="12d1f55f20b117d559cc13e02d487ce35435ed6b0ccc37be0532918aaf2f6cc6" Jul 06 20:32:37.175070 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:37.175039 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:37.182453 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:37.182422 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v2d5c"] Jul 06 20:32:38.559632 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:38.559598 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ca023a-1c80-4812-bb0e-4dc4a871932c" path="/var/lib/kubelet/pods/52ca023a-1c80-4812-bb0e-4dc4a871932c/volumes" Jul 06 20:32:51.539852 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.539816 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6"] Jul 06 20:32:51.540472 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.540452 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca023a-1c80-4812-bb0e-4dc4a871932c" containerName="manager" Jul 06 20:32:51.540521 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.540474 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca023a-1c80-4812-bb0e-4dc4a871932c" containerName="manager" Jul 06 20:32:51.540564 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.540540 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca023a-1c80-4812-bb0e-4dc4a871932c" containerName="manager" Jul 06 20:32:51.542827 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.542806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.545532 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.545504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8l8r9\"" Jul 06 20:32:51.566467 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.566441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6"] Jul 06 20:32:51.686849 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.686815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.686866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.686896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.686918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.686959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687017 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.687011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687237 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.687029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687237 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.687051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdsb\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-kube-api-access-jrdsb\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.687237 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.687083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788392 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788392 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdsb\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-kube-api-access-jrdsb\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.788728 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.788706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.789473 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.789435 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.789605 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.789481 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.789605 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.789560 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.789605 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.789578 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.789781 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.789654 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.790907 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.790856 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.791090 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.791073 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.796573 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.796552 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.796771 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.796753 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdsb\" (UniqueName: \"kubernetes.io/projected/d1ff50c0-9de3-4711-9843-e2ed67efcec2-kube-api-access-jrdsb\") pod \"maas-default-gateway-openshift-default-56654dcf7-cl6m6\" (UID: \"d1ff50c0-9de3-4711-9843-e2ed67efcec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:51.853989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:51.853954 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:52.016190 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.016141 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6"] Jul 06 20:32:52.017676 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:32:52.017647 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ff50c0_9de3_4711_9843_e2ed67efcec2.slice/crio-3dd5bd9387f1cf051284c1ba423816c6a6beaa896814c2835d48e35ee78d7282 WatchSource:0}: Error finding container 3dd5bd9387f1cf051284c1ba423816c6a6beaa896814c2835d48e35ee78d7282: Status 404 returned error can't find the container with id 3dd5bd9387f1cf051284c1ba423816c6a6beaa896814c2835d48e35ee78d7282 Jul 06 20:32:52.019741 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.019709 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:32:52.019838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.019792 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:32:52.019902 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.019838 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:32:52.237996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.237954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" event={"ID":"d1ff50c0-9de3-4711-9843-e2ed67efcec2","Type":"ContainerStarted","Data":"7831771447f593f90a12dcd825136ccf40c842e1a953e45cfc05701e27c007da"} Jul 06 20:32:52.238204 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.238022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" event={"ID":"d1ff50c0-9de3-4711-9843-e2ed67efcec2","Type":"ContainerStarted","Data":"3dd5bd9387f1cf051284c1ba423816c6a6beaa896814c2835d48e35ee78d7282"} Jul 06 20:32:52.259501 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.259440 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podStartSLOduration=1.259417948 podStartE2EDuration="1.259417948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:52.258714465 +0000 UTC m=+514.799770904" watchObservedRunningTime="2026-07-06 20:32:52.259417948 +0000 UTC m=+514.800474388" Jul 06 20:32:52.854266 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:52.854221 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:53.859858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:53.859826 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:54.240548 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.240444 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:32:54.244557 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.244533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:32:54.246734 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.246712 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4bdzw\"" Jul 06 20:32:54.250669 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.250645 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:54.251767 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.251743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:32:54.252938 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.252914 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:32:54.414238 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.414191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzsgh\" (UniqueName: \"kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh\") pod \"authorino-58cd955675-cgv5k\" (UID: \"cc9ff3ce-43ed-4b09-993d-284482284380\") " pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:32:54.515198 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.515083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzsgh\" (UniqueName: \"kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh\") pod \"authorino-58cd955675-cgv5k\" (UID: \"cc9ff3ce-43ed-4b09-993d-284482284380\") " pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:32:54.523916 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.523889 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzsgh\" (UniqueName: \"kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh\") pod \"authorino-58cd955675-cgv5k\" (UID: \"cc9ff3ce-43ed-4b09-993d-284482284380\") " pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:32:54.556327 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.556278 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:32:54.702965 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:54.702933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:32:54.704973 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:32:54.704938 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc9ff3ce_43ed_4b09_993d_284482284380.slice/crio-fdc872833fd5ae7e01783abfc4a132b041b379d5a6fba40f2fc65c6487bdf740 WatchSource:0}: Error finding container fdc872833fd5ae7e01783abfc4a132b041b379d5a6fba40f2fc65c6487bdf740: Status 404 returned error can't find the container with id fdc872833fd5ae7e01783abfc4a132b041b379d5a6fba40f2fc65c6487bdf740 Jul 06 20:32:55.199632 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.199594 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-688f6cc798-h2bj7" podUID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" containerName="console" containerID="cri-o://fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da" gracePeriod=15 Jul 06 20:32:55.258011 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.257975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgv5k" event={"ID":"cc9ff3ce-43ed-4b09-993d-284482284380","Type":"ContainerStarted","Data":"fdc872833fd5ae7e01783abfc4a132b041b379d5a6fba40f2fc65c6487bdf740"} Jul 06 20:32:55.467965 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.467936 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:32:55.624552 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624510 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624569 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624638 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624670 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624769 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2ll\" (UniqueName: \"kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.624972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624818 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert\") pod \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\" (UID: \"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06\") " Jul 06 20:32:55.625073 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.624974 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:55.625299 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.625274 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-trusted-ca-bundle\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.625398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.625324 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:55.625398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.625349 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:55.625398 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.625379 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config" (OuterVolumeSpecName: "console-config") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:55.627488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.627465 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:32:55.627945 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.627912 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll" (OuterVolumeSpecName: "kube-api-access-6w2ll") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "kube-api-access-6w2ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:55.627945 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.627916 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" (UID: "c4fb3ad8-552c-4f30-8ca7-42e9da2fab06"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726769 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-service-ca\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726803 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726818 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-oauth-config\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726834 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2ll\" (UniqueName: \"kubernetes.io/projected/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-kube-api-access-6w2ll\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726847 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-console-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:55.727565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:55.726860 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06-oauth-serving-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:32:56.265489 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.265453 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" containerID="fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da" exitCode=0 Jul 06 20:32:56.265964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.265503 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688f6cc798-h2bj7" Jul 06 20:32:56.265964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.265531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688f6cc798-h2bj7" event={"ID":"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06","Type":"ContainerDied","Data":"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da"} Jul 06 20:32:56.265964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.265583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688f6cc798-h2bj7" event={"ID":"c4fb3ad8-552c-4f30-8ca7-42e9da2fab06","Type":"ContainerDied","Data":"0d0214349f326fb2ddf2bac0a84b2a75769eab262cdf1f40555b2189dd893fbb"} Jul 06 20:32:56.265964 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.265604 2562 scope.go:117] "RemoveContainer" containerID="fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da" Jul 06 20:32:56.275211 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.275194 2562 scope.go:117] "RemoveContainer" containerID="fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da" Jul 06 20:32:56.275442 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:32:56.275427 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da\": container with ID starting with fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da not found: ID does not exist" containerID="fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da" Jul 06 20:32:56.275487 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.275450 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da"} err="failed to get container status \"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da\": rpc error: code = NotFound desc = could not find container \"fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da\": container with ID starting with fcc417badb1eb93fda4984239d2087ed4148aefc97fbfb66105188ec26d586da not found: ID does not exist" Jul 06 20:32:56.289434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.289407 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:32:56.297405 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.297376 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-688f6cc798-h2bj7"] Jul 06 20:32:56.562666 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:56.562629 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" path="/var/lib/kubelet/pods/c4fb3ad8-552c-4f30-8ca7-42e9da2fab06/volumes" Jul 06 20:32:58.281950 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:58.281907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgv5k" event={"ID":"cc9ff3ce-43ed-4b09-993d-284482284380","Type":"ContainerStarted","Data":"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9"} Jul 06 20:32:58.301419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:58.301360 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cgv5k" podStartSLOduration=0.867768711 podStartE2EDuration="4.301344786s" totalImagesPullingTime="3.433576075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:32:58.299789098 +0000 UTC m=+520.840845549" watchObservedRunningTime="2026-07-06 20:32:58.301344786 +0000 UTC m=+520.842401222" Jul 06 20:32:58.850503 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:32:58.850468 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:33:00.293573 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:00.293514 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cgv5k" podUID="cc9ff3ce-43ed-4b09-993d-284482284380" containerName="authorino" containerID="cri-o://df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9" gracePeriod=30 Jul 06 20:33:00.545868 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:00.545792 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:33:00.676906 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:00.676866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzsgh\" (UniqueName: \"kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh\") pod \"cc9ff3ce-43ed-4b09-993d-284482284380\" (UID: \"cc9ff3ce-43ed-4b09-993d-284482284380\") " Jul 06 20:33:00.678940 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:00.678915 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh" (OuterVolumeSpecName: "kube-api-access-gzsgh") pod "cc9ff3ce-43ed-4b09-993d-284482284380" (UID: "cc9ff3ce-43ed-4b09-993d-284482284380"). InnerVolumeSpecName "kube-api-access-gzsgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:33:00.778436 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:00.778403 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzsgh\" (UniqueName: \"kubernetes.io/projected/cc9ff3ce-43ed-4b09-993d-284482284380-kube-api-access-gzsgh\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:01.300520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.300483 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc9ff3ce-43ed-4b09-993d-284482284380" containerID="df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9" exitCode=0 Jul 06 20:33:01.300988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.300526 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgv5k" Jul 06 20:33:01.300988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.300565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgv5k" event={"ID":"cc9ff3ce-43ed-4b09-993d-284482284380","Type":"ContainerDied","Data":"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9"} Jul 06 20:33:01.300988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.300606 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgv5k" event={"ID":"cc9ff3ce-43ed-4b09-993d-284482284380","Type":"ContainerDied","Data":"fdc872833fd5ae7e01783abfc4a132b041b379d5a6fba40f2fc65c6487bdf740"} Jul 06 20:33:01.300988 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.300630 2562 scope.go:117] "RemoveContainer" containerID="df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9" Jul 06 20:33:01.309882 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.309862 2562 scope.go:117] "RemoveContainer" containerID="df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9" Jul 06 20:33:01.310187 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:33:01.310167 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9\": container with ID starting with df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9 not found: ID does not exist" containerID="df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9" Jul 06 20:33:01.310269 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.310200 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9"} err="failed to get container status \"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9\": rpc error: code = NotFound desc = could not find container \"df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9\": container with ID starting with df5f0355677567fae80b61983b8e975e8c5cf4e3bd8b541ce2a95e77c0acd2d9 not found: ID does not exist" Jul 06 20:33:01.321366 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.321336 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:33:01.325140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:01.325102 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgv5k"] Jul 06 20:33:02.559352 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:02.559317 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9ff3ce-43ed-4b09-993d-284482284380" path="/var/lib/kubelet/pods/cc9ff3ce-43ed-4b09-993d-284482284380/volumes" Jul 06 20:33:37.349207 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349168 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twg2d"] Jul 06 20:33:37.349808 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349786 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" containerName="console" Jul 06 20:33:37.349900 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349812 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" containerName="console" Jul 06 20:33:37.349900 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349826 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ff3ce-43ed-4b09-993d-284482284380" containerName="authorino" Jul 06 20:33:37.349900 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349834 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ff3ce-43ed-4b09-993d-284482284380" containerName="authorino" Jul 06 20:33:37.350064 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349964 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ff3ce-43ed-4b09-993d-284482284380" containerName="authorino" Jul 06 20:33:37.350064 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.349985 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fb3ad8-552c-4f30-8ca7-42e9da2fab06" containerName="console" Jul 06 20:33:37.356908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.356882 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:37.357868 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.357839 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twg2d"] Jul 06 20:33:37.359401 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.359375 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4bdzw\"" Jul 06 20:33:37.504088 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.504049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mdj\" (UniqueName: \"kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj\") pod \"authorino-5d4b67c446-twg2d\" (UID: \"cc078aa5-d581-47cb-894d-8f8212c9bf3a\") " pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:37.580525 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.580487 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twg2d"] Jul 06 20:33:37.580818 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:33:37.580799 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t4mdj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-twg2d" podUID="cc078aa5-d581-47cb-894d-8f8212c9bf3a" Jul 06 20:33:37.605230 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.605117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mdj\" (UniqueName: \"kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj\") pod \"authorino-5d4b67c446-twg2d\" (UID: \"cc078aa5-d581-47cb-894d-8f8212c9bf3a\") " pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:37.607563 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.607542 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fcbb76d44-cnlfz"] Jul 06 20:33:37.609986 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.609972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:37.620229 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.620202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcbb76d44-cnlfz"] Jul 06 20:33:37.621068 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.621049 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mdj\" (UniqueName: \"kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj\") pod \"authorino-5d4b67c446-twg2d\" (UID: \"cc078aa5-d581-47cb-894d-8f8212c9bf3a\") " pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:37.650927 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.650889 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcbb76d44-cnlfz"] Jul 06 20:33:37.651212 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:33:37.651191 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l4zrr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" podUID="17bc62f7-f412-4e77-a383-5001d2b83ec7" Jul 06 20:33:37.680332 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.680293 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:33:37.682881 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.682861 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.685060 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.685044 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 20:33:37.690943 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.690918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:33:37.706407 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.706375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zrr\" (UniqueName: \"kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr\") pod \"authorino-5fcbb76d44-cnlfz\" (UID: \"17bc62f7-f412-4e77-a383-5001d2b83ec7\") " pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:37.807864 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.807833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbr2w\" (UniqueName: \"kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.808065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.807901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zrr\" (UniqueName: \"kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr\") pod \"authorino-5fcbb76d44-cnlfz\" (UID: \"17bc62f7-f412-4e77-a383-5001d2b83ec7\") " pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:37.808065 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.807941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.815774 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.815745 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zrr\" (UniqueName: \"kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr\") pod \"authorino-5fcbb76d44-cnlfz\" (UID: \"17bc62f7-f412-4e77-a383-5001d2b83ec7\") " pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:37.909484 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.909385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbr2w\" (UniqueName: \"kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.909484 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.909468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.912015 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.911991 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.917663 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.917640 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbr2w\" (UniqueName: \"kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w\") pod \"authorino-57f48df4fb-qfbvj\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:37.993812 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:37.993778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:33:38.131524 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.131499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:33:38.133306 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:33:38.133280 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod053402bc_beaf_4ec3_9ce5_2dc0978fff87.slice/crio-1bd329b6e81f590cfda0bd806ae05c32709f5594b5484cd6fe6bc8acc639308b WatchSource:0}: Error finding container 1bd329b6e81f590cfda0bd806ae05c32709f5594b5484cd6fe6bc8acc639308b: Status 404 returned error can't find the container with id 1bd329b6e81f590cfda0bd806ae05c32709f5594b5484cd6fe6bc8acc639308b Jul 06 20:33:38.519377 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.519342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:38.519815 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.519375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" event={"ID":"053402bc-beaf-4ec3-9ce5-2dc0978fff87","Type":"ContainerStarted","Data":"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd"} Jul 06 20:33:38.519815 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.519416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" event={"ID":"053402bc-beaf-4ec3-9ce5-2dc0978fff87","Type":"ContainerStarted","Data":"1bd329b6e81f590cfda0bd806ae05c32709f5594b5484cd6fe6bc8acc639308b"} Jul 06 20:33:38.519815 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.519439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:38.525950 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.525915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:38.529689 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.529662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:38.555265 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.555121 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" podStartSLOduration=1.263088984 podStartE2EDuration="1.555099862s" totalImagesPullingTime="292.010878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:38.551693193 +0000 UTC m=+561.092749630" watchObservedRunningTime="2026-07-06 20:33:38.555099862 +0000 UTC m=+561.096156300" Jul 06 20:33:38.614960 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.614927 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4mdj\" (UniqueName: \"kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj\") pod \"cc078aa5-d581-47cb-894d-8f8212c9bf3a\" (UID: \"cc078aa5-d581-47cb-894d-8f8212c9bf3a\") " Jul 06 20:33:38.615140 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.615031 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4zrr\" (UniqueName: \"kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr\") pod \"17bc62f7-f412-4e77-a383-5001d2b83ec7\" (UID: \"17bc62f7-f412-4e77-a383-5001d2b83ec7\") " Jul 06 20:33:38.617101 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.617076 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj" (OuterVolumeSpecName: "kube-api-access-t4mdj") pod "cc078aa5-d581-47cb-894d-8f8212c9bf3a" (UID: "cc078aa5-d581-47cb-894d-8f8212c9bf3a"). InnerVolumeSpecName "kube-api-access-t4mdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:33:38.617101 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.617087 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr" (OuterVolumeSpecName: "kube-api-access-l4zrr") pod "17bc62f7-f412-4e77-a383-5001d2b83ec7" (UID: "17bc62f7-f412-4e77-a383-5001d2b83ec7"). InnerVolumeSpecName "kube-api-access-l4zrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:33:38.716661 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.716552 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mdj\" (UniqueName: \"kubernetes.io/projected/cc078aa5-d581-47cb-894d-8f8212c9bf3a-kube-api-access-t4mdj\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:38.716661 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:38.716588 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zrr\" (UniqueName: \"kubernetes.io/projected/17bc62f7-f412-4e77-a383-5001d2b83ec7-kube-api-access-l4zrr\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:39.531186 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.525933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twg2d" Jul 06 20:33:39.531186 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.527393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcbb76d44-cnlfz" Jul 06 20:33:39.559031 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.558998 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twg2d"] Jul 06 20:33:39.562887 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.562856 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twg2d"] Jul 06 20:33:39.585205 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.585143 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcbb76d44-cnlfz"] Jul 06 20:33:39.589428 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:39.589395 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fcbb76d44-cnlfz"] Jul 06 20:33:40.559917 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:40.559881 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17bc62f7-f412-4e77-a383-5001d2b83ec7" path="/var/lib/kubelet/pods/17bc62f7-f412-4e77-a383-5001d2b83ec7/volumes" Jul 06 20:33:40.560339 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:40.560115 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc078aa5-d581-47cb-894d-8f8212c9bf3a" path="/var/lib/kubelet/pods/cc078aa5-d581-47cb-894d-8f8212c9bf3a/volumes" Jul 06 20:33:43.056847 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.056804 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:33:43.060128 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.060102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.062747 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.062721 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sdpss\"" Jul 06 20:33:43.062989 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.062972 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:33:43.063825 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.063787 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:33:43.076657 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.076629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:33:43.159721 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.159675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcnj\" (UniqueName: \"kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.159899 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.159739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.159899 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.159831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.260819 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.260781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.260995 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.260865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcnj\" (UniqueName: \"kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.260995 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.260909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.261639 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.261619 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.262630 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.262608 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:33:43.263457 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.263433 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.266316 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.266291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.272277 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.272254 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcnj\" (UniqueName: \"kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj\") pod \"maas-controller-7d4c849f79-5xlcj\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.277400 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.277378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:33:43.362366 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.362319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpz2\" (UniqueName: \"kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.362570 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.362432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.362570 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.362502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.372250 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.372220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:43.463841 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.463809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpz2\" (UniqueName: \"kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.464000 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.463915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.464061 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.464019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.464599 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.464572 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.466436 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.466414 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.472853 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.472828 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpz2\" (UniqueName: \"kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2\") pod \"maas-controller-86785d597b-wgxjh\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.520874 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.520842 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:33:43.522841 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:33:43.522809 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fd41f3_e8ae_4150_b6ed_f7f8498df80c.slice/crio-248a3a592d5d692e416c315b453de3f03903eada980d7cce01ff921f8b7487c3 WatchSource:0}: Error finding container 248a3a592d5d692e416c315b453de3f03903eada980d7cce01ff921f8b7487c3: Status 404 returned error can't find the container with id 248a3a592d5d692e416c315b453de3f03903eada980d7cce01ff921f8b7487c3 Jul 06 20:33:43.552693 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.552656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" event={"ID":"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c","Type":"ContainerStarted","Data":"248a3a592d5d692e416c315b453de3f03903eada980d7cce01ff921f8b7487c3"} Jul 06 20:33:43.591270 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.591236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:43.733879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:43.733843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:33:43.735323 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:33:43.735296 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2f6682_62bf_433d_be6a_f6a33d1eb4d8.slice/crio-f29d9c2891b05e6a0cd08349c1599d1c7608a588da766dd22a0f28f99254cdcc WatchSource:0}: Error finding container f29d9c2891b05e6a0cd08349c1599d1c7608a588da766dd22a0f28f99254cdcc: Status 404 returned error can't find the container with id f29d9c2891b05e6a0cd08349c1599d1c7608a588da766dd22a0f28f99254cdcc Jul 06 20:33:44.573417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:44.573326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-wgxjh" event={"ID":"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8","Type":"ContainerStarted","Data":"f29d9c2891b05e6a0cd08349c1599d1c7608a588da766dd22a0f28f99254cdcc"} Jul 06 20:33:46.587413 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.587366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-wgxjh" event={"ID":"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8","Type":"ContainerStarted","Data":"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c"} Jul 06 20:33:46.587867 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.587522 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:46.588972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.588943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" event={"ID":"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c","Type":"ContainerStarted","Data":"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db"} Jul 06 20:33:46.589091 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.589002 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:46.616716 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.616325 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86785d597b-wgxjh" podStartSLOduration=1.5458320429999999 podStartE2EDuration="3.616308166s" totalImagesPullingTime="2.070476123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:46.613192075 +0000 UTC m=+569.154248511" watchObservedRunningTime="2026-07-06 20:33:46.616308166 +0000 UTC m=+569.157364604" Jul 06 20:33:46.639163 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:46.639093 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" podStartSLOduration=1.367276667 podStartE2EDuration="3.639075417s" totalImagesPullingTime="2.27179875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:46.637487231 +0000 UTC m=+569.178543668" watchObservedRunningTime="2026-07-06 20:33:46.639075417 +0000 UTC m=+569.180131853" Jul 06 20:33:57.601700 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:57.601669 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:57.602218 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:57.602183 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:33:57.665093 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:57.665058 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:33:57.665346 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:57.665322 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" podUID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" containerName="manager" containerID="cri-o://8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db" gracePeriod=30 Jul 06 20:33:57.922979 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:57.922952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:58.002053 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.002018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca\") pod \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " Jul 06 20:33:58.002276 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.002105 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert\") pod \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " Jul 06 20:33:58.002276 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.002183 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfcnj\" (UniqueName: \"kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj\") pod \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\" (UID: \"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c\") " Jul 06 20:33:58.002520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.002491 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" (UID: "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:33:58.004434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.004413 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj" (OuterVolumeSpecName: "kube-api-access-jfcnj") pod "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" (UID: "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c"). InnerVolumeSpecName "kube-api-access-jfcnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:33:58.004434 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.004411 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" (UID: "a1fd41f3-e8ae-4150-b6ed-f7f8498df80c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:33:58.103565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.103471 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-service-ca\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:58.103565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.103504 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-webhook-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:58.103565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.103513 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcnj\" (UniqueName: \"kubernetes.io/projected/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c-kube-api-access-jfcnj\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:33:58.664746 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.664710 2562 generic.go:356] "Generic (PLEG): container finished" podID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" containerID="8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db" exitCode=0 Jul 06 20:33:58.665194 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.664759 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" Jul 06 20:33:58.665194 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.664756 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" event={"ID":"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c","Type":"ContainerDied","Data":"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db"} Jul 06 20:33:58.665194 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.664879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d4c849f79-5xlcj" event={"ID":"a1fd41f3-e8ae-4150-b6ed-f7f8498df80c","Type":"ContainerDied","Data":"248a3a592d5d692e416c315b453de3f03903eada980d7cce01ff921f8b7487c3"} Jul 06 20:33:58.665194 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.664909 2562 scope.go:117] "RemoveContainer" containerID="8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db" Jul 06 20:33:58.674302 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.674285 2562 scope.go:117] "RemoveContainer" containerID="8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db" Jul 06 20:33:58.674577 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:33:58.674558 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db\": container with ID starting with 8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db not found: ID does not exist" containerID="8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db" Jul 06 20:33:58.674628 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.674588 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db"} err="failed to get container status \"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db\": rpc error: code = NotFound desc = could not find container \"8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db\": container with ID starting with 8f37b9549e4f9a11acfbfc53f1dc2b7e1c75b9ac1c1fc5e3042f3dd20273a3db not found: ID does not exist" Jul 06 20:33:58.683377 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.683355 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:33:58.688814 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:33:58.688794 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d4c849f79-5xlcj"] Jul 06 20:34:00.559462 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:00.559428 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" path="/var/lib/kubelet/pods/a1fd41f3-e8ae-4150-b6ed-f7f8498df80c/volumes" Jul 06 20:34:17.647532 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.647493 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-7f47dc944b-mvpj9"] Jul 06 20:34:17.648488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.648456 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" containerName="manager" Jul 06 20:34:17.648488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.648490 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" containerName="manager" Jul 06 20:34:17.648895 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.648880 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1fd41f3-e8ae-4150-b6ed-f7f8498df80c" containerName="manager" Jul 06 20:34:17.652209 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.652185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.656070 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.656043 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 20:34:17.656212 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.656166 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vp9nq\"" Jul 06 20:34:17.670028 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.669951 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7f47dc944b-mvpj9"] Jul 06 20:34:17.786494 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.786458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a035d24-0aa4-4de0-9a86-3389f20d6e64-openshift-service-ca\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.786680 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.786529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrb7\" (UniqueName: \"kubernetes.io/projected/0a035d24-0aa4-4de0-9a86-3389f20d6e64-kube-api-access-tcrb7\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.786680 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.786612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.887130 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.887095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.887336 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.887192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a035d24-0aa4-4de0-9a86-3389f20d6e64-openshift-service-ca\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.887336 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.887269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrb7\" (UniqueName: \"kubernetes.io/projected/0a035d24-0aa4-4de0-9a86-3389f20d6e64-kube-api-access-tcrb7\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.887336 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:34:17.887270 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 06 20:34:17.887512 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:34:17.887403 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls podName:0a035d24-0aa4-4de0-9a86-3389f20d6e64 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:18.387378967 +0000 UTC m=+600.928435382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls") pod "maas-api-7f47dc944b-mvpj9" (UID: "0a035d24-0aa4-4de0-9a86-3389f20d6e64") : secret "maas-api-serving-cert" not found Jul 06 20:34:17.888203 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.888177 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a035d24-0aa4-4de0-9a86-3389f20d6e64-openshift-service-ca\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:17.896523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:17.896495 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrb7\" (UniqueName: \"kubernetes.io/projected/0a035d24-0aa4-4de0-9a86-3389f20d6e64-kube-api-access-tcrb7\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:18.261760 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.261720 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xtq9t"] Jul 06 20:34:18.264419 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.264402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.267453 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.267435 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 20:34:18.267520 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.267435 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtn9v\"" Jul 06 20:34:18.277575 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.277547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xtq9t"] Jul 06 20:34:18.391480 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.391442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:18.391688 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.391502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4m94\" (UniqueName: \"kubernetes.io/projected/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-kube-api-access-r4m94\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.391688 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.391552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.394180 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.394133 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a035d24-0aa4-4de0-9a86-3389f20d6e64-maas-api-tls\") pod \"maas-api-7f47dc944b-mvpj9\" (UID: \"0a035d24-0aa4-4de0-9a86-3389f20d6e64\") " pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:18.449424 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.449396 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:34:18.450333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.450309 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:34:18.492681 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.492570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4m94\" (UniqueName: \"kubernetes.io/projected/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-kube-api-access-r4m94\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.492681 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.492668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.504201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.495937 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 20:34:18.504313 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.504290 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.506928 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.506901 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4m94\" (UniqueName: \"kubernetes.io/projected/8a5b0f6c-21ab-43ac-a5af-591fe8c6292a-kube-api-access-r4m94\") pod \"payload-pre-processing-7964457696-xtq9t\" (UID: \"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a\") " pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.567456 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.567427 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vp9nq\"" Jul 06 20:34:18.576173 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.576118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:18.578692 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.578673 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtn9v\"" Jul 06 20:34:18.586625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.586595 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:18.782297 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.782222 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7f47dc944b-mvpj9"] Jul 06 20:34:18.783981 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:34:18.783315 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a035d24_0aa4_4de0_9a86_3389f20d6e64.slice/crio-bc46daa67224fb2c935594003f57442ddc712c3d197c5e147cf2485a25312541 WatchSource:0}: Error finding container bc46daa67224fb2c935594003f57442ddc712c3d197c5e147cf2485a25312541: Status 404 returned error can't find the container with id bc46daa67224fb2c935594003f57442ddc712c3d197c5e147cf2485a25312541 Jul 06 20:34:18.825051 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:18.825017 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-xtq9t"] Jul 06 20:34:18.825468 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:34:18.825436 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5b0f6c_21ab_43ac_a5af_591fe8c6292a.slice/crio-d460a7db335a9083e18efb206a2bdb59b13ebb6f787ae3419fba434e829b9245 WatchSource:0}: Error finding container d460a7db335a9083e18efb206a2bdb59b13ebb6f787ae3419fba434e829b9245: Status 404 returned error can't find the container with id d460a7db335a9083e18efb206a2bdb59b13ebb6f787ae3419fba434e829b9245 Jul 06 20:34:19.792567 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:19.792521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" event={"ID":"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a","Type":"ContainerStarted","Data":"d460a7db335a9083e18efb206a2bdb59b13ebb6f787ae3419fba434e829b9245"} Jul 06 20:34:19.793905 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:19.793873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7f47dc944b-mvpj9" event={"ID":"0a035d24-0aa4-4de0-9a86-3389f20d6e64","Type":"ContainerStarted","Data":"bc46daa67224fb2c935594003f57442ddc712c3d197c5e147cf2485a25312541"} Jul 06 20:34:21.833924 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.833883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" event={"ID":"8a5b0f6c-21ab-43ac-a5af-591fe8c6292a","Type":"ContainerStarted","Data":"f5db2abb216affb63badb2b40de9e449e62b729bdf72dd5d276c3824e20ebbca"} Jul 06 20:34:21.834453 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.834426 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:21.835554 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.835530 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7f47dc944b-mvpj9" event={"ID":"0a035d24-0aa4-4de0-9a86-3389f20d6e64","Type":"ContainerStarted","Data":"1072e6bedd7eed07cd2826d375915e7268ecb6e43e2b29ce0f86043c5e10ba6c"} Jul 06 20:34:21.835663 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.835577 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:21.852338 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.852287 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" podStartSLOduration=1.799525471 podStartE2EDuration="3.852270133s" totalImagesPullingTime="2.052744662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:21.851352573 +0000 UTC m=+604.392409013" watchObservedRunningTime="2026-07-06 20:34:21.852270133 +0000 UTC m=+604.393326570" Jul 06 20:34:21.876515 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:21.876468 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-7f47dc944b-mvpj9" podStartSLOduration=2.778976174 podStartE2EDuration="4.876454402s" totalImagesPullingTime="2.097478228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:21.8730783 +0000 UTC m=+604.414134737" watchObservedRunningTime="2026-07-06 20:34:21.876454402 +0000 UTC m=+604.417510839" Jul 06 20:34:25.292363 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:25.292326 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:34:25.292764 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:25.292389 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podUID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:34:26.870010 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.869963 2562 generic.go:356] "Generic (PLEG): container finished" podID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerID="7831771447f593f90a12dcd825136ccf40c842e1a953e45cfc05701e27c007da" exitCode=0 Jul 06 20:34:26.870468 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.870030 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" event={"ID":"d1ff50c0-9de3-4711-9843-e2ed67efcec2","Type":"ContainerDied","Data":"7831771447f593f90a12dcd825136ccf40c842e1a953e45cfc05701e27c007da"} Jul 06 20:34:26.870468 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.870441 2562 scope.go:117] "RemoveContainer" containerID="7831771447f593f90a12dcd825136ccf40c842e1a953e45cfc05701e27c007da" Jul 06 20:34:26.871475 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.871448 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:34:26.871577 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.871515 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:34:26.871577 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:26.871545 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:34:27.847492 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:27.847462 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-7f47dc944b-mvpj9" Jul 06 20:34:27.878723 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:27.878687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" event={"ID":"d1ff50c0-9de3-4711-9843-e2ed67efcec2","Type":"ContainerStarted","Data":"9d814dc9350624ba5eb0da84590eb94b6da3ff739ae93a0b8a5e888cb691fa1e"} Jul 06 20:34:28.854681 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:28.854637 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:34:28.854956 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:28.854893 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:34:28.854956 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:28.854935 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podUID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 20:34:29.854849 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:29.854812 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:34:29.855301 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:29.854885 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podUID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 20:34:30.854707 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:30.854667 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:34:30.854929 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:30.854749 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podUID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 20:34:31.855308 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:31.855272 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cl6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 20:34:31.855726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:31.855333 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" podUID="d1ff50c0-9de3-4711-9843-e2ed67efcec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 20:34:32.857682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:32.857651 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:34:32.858090 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:32.858012 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:34:32.858725 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:32.858707 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cl6m6" Jul 06 20:34:33.849829 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:33.849795 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-xtq9t" Jul 06 20:34:58.319386 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.319349 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc"] Jul 06 20:34:58.324539 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.324521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.328615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.328584 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 20:34:58.328757 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.328592 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 20:34:58.328757 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.328594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 20:34:58.328757 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.328594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-25gn5\"" Jul 06 20:34:58.349860 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.349824 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc"] Jul 06 20:34:58.405241 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22f80af-fe78-4286-a15d-8fb510825835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.405426 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.405426 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.405426 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.405426 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54tv\" (UniqueName: \"kubernetes.io/projected/d22f80af-fe78-4286-a15d-8fb510825835-kube-api-access-g54tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.405574 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.405426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506001 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.505957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506001 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506304 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506304 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54tv\" (UniqueName: \"kubernetes.io/projected/d22f80af-fe78-4286-a15d-8fb510825835-kube-api-access-g54tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506304 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506475 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22f80af-fe78-4286-a15d-8fb510825835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506566 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506542 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506622 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506560 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.506660 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.506626 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.508373 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.508344 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d22f80af-fe78-4286-a15d-8fb510825835-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.508889 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.508868 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22f80af-fe78-4286-a15d-8fb510825835-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.514779 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.514754 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54tv\" (UniqueName: \"kubernetes.io/projected/d22f80af-fe78-4286-a15d-8fb510825835-kube-api-access-g54tv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhbfc\" (UID: \"d22f80af-fe78-4286-a15d-8fb510825835\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.636009 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.635904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:34:58.778576 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.778546 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc"] Jul 06 20:34:58.779989 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:34:58.779955 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22f80af_fe78_4286_a15d_8fb510825835.slice/crio-04cfd39d8a70ac21426e16bdb84cb4a9313b230f18137bc191a4cb742f7a40bc WatchSource:0}: Error finding container 04cfd39d8a70ac21426e16bdb84cb4a9313b230f18137bc191a4cb742f7a40bc: Status 404 returned error can't find the container with id 04cfd39d8a70ac21426e16bdb84cb4a9313b230f18137bc191a4cb742f7a40bc Jul 06 20:34:58.781304 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:58.781283 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:34:59.074337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:34:59.074253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" event={"ID":"d22f80af-fe78-4286-a15d-8fb510825835","Type":"ContainerStarted","Data":"04cfd39d8a70ac21426e16bdb84cb4a9313b230f18137bc191a4cb742f7a40bc"} Jul 06 20:35:01.087904 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:01.087861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" event={"ID":"d22f80af-fe78-4286-a15d-8fb510825835","Type":"ContainerStarted","Data":"c5a50131e7b9b82ea18b01396cce834fb463019f0d3d1a04ecfaaed5d9dd6689"} Jul 06 20:35:01.107496 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:01.107441 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" podStartSLOduration=1.8634358039999999 podStartE2EDuration="3.107424712s" totalImagesPullingTime="1.243988908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:01.105115442 +0000 UTC m=+643.646171881" watchObservedRunningTime="2026-07-06 20:35:01.107424712 +0000 UTC m=+643.648481148" Jul 06 20:35:08.636839 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:08.636810 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:35:08.637278 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:08.637061 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:35:08.649602 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:08.649572 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:35:09.152330 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:09.152291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhbfc" Jul 06 20:35:42.361102 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.361064 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz"] Jul 06 20:35:42.365494 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.365466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.368036 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.368007 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 20:35:42.383102 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.383066 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz"] Jul 06 20:35:42.540437 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.540625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7265a401-2fa6-4319-832e-b8bef7895592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.540625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.540625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.540625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.540625 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.540592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws7sw\" (UniqueName: \"kubernetes.io/projected/7265a401-2fa6-4319-832e-b8bef7895592-kube-api-access-ws7sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642059 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.641964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7265a401-2fa6-4319-832e-b8bef7895592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642059 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642059 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642397 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642397 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws7sw\" (UniqueName: \"kubernetes.io/projected/7265a401-2fa6-4319-832e-b8bef7895592-kube-api-access-ws7sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642397 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642568 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642612 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642580 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.642713 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.642640 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.644442 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.644419 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7265a401-2fa6-4319-832e-b8bef7895592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.644661 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.644645 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7265a401-2fa6-4319-832e-b8bef7895592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.651496 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.651474 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws7sw\" (UniqueName: \"kubernetes.io/projected/7265a401-2fa6-4319-832e-b8bef7895592-kube-api-access-ws7sw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lrjnz\" (UID: \"7265a401-2fa6-4319-832e-b8bef7895592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.677528 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.677493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:42.824112 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:42.824077 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz"] Jul 06 20:35:42.824755 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:35:42.824726 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7265a401_2fa6_4319_832e_b8bef7895592.slice/crio-63c058bcf6f8689d6c5d6d88962f7032f0540ed87bdfae9e1d0665e565920d43 WatchSource:0}: Error finding container 63c058bcf6f8689d6c5d6d88962f7032f0540ed87bdfae9e1d0665e565920d43: Status 404 returned error can't find the container with id 63c058bcf6f8689d6c5d6d88962f7032f0540ed87bdfae9e1d0665e565920d43 Jul 06 20:35:43.355663 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:43.355624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" event={"ID":"7265a401-2fa6-4319-832e-b8bef7895592","Type":"ContainerStarted","Data":"55fa7fc2b8c80f5b7a06e5f7f4055abdc1f49c37cfa2ac506581f03f0f2f9394"} Jul 06 20:35:43.355846 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:43.355682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" event={"ID":"7265a401-2fa6-4319-832e-b8bef7895592","Type":"ContainerStarted","Data":"63c058bcf6f8689d6c5d6d88962f7032f0540ed87bdfae9e1d0665e565920d43"} Jul 06 20:35:43.375713 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:43.375657 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" podStartSLOduration=1.177217412 podStartE2EDuration="1.375639903s" totalImagesPullingTime="198.422491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:43.373613063 +0000 UTC m=+685.914669500" watchObservedRunningTime="2026-07-06 20:35:43.375639903 +0000 UTC m=+685.916696339" Jul 06 20:35:52.677785 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:52.677747 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:52.678242 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:52.677813 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:52.690733 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:52.690708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:35:53.432633 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:35:53.432589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lrjnz" Jul 06 20:36:45.957192 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:45.957056 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:36:45.957785 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:45.957363 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" podUID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" containerName="authorino" containerID="cri-o://3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd" gracePeriod=30 Jul 06 20:36:46.231274 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.231243 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:36:46.269732 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.269628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert\") pod \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " Jul 06 20:36:46.269957 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.269755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbr2w\" (UniqueName: \"kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w\") pod \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\" (UID: \"053402bc-beaf-4ec3-9ce5-2dc0978fff87\") " Jul 06 20:36:46.272503 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.272302 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w" (OuterVolumeSpecName: "kube-api-access-cbr2w") pod "053402bc-beaf-4ec3-9ce5-2dc0978fff87" (UID: "053402bc-beaf-4ec3-9ce5-2dc0978fff87"). InnerVolumeSpecName "kube-api-access-cbr2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:36:46.284314 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.284276 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "053402bc-beaf-4ec3-9ce5-2dc0978fff87" (UID: "053402bc-beaf-4ec3-9ce5-2dc0978fff87"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:36:46.371376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.371336 2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/053402bc-beaf-4ec3-9ce5-2dc0978fff87-tls-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:36:46.371376 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.371370 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbr2w\" (UniqueName: \"kubernetes.io/projected/053402bc-beaf-4ec3-9ce5-2dc0978fff87-kube-api-access-cbr2w\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:36:46.759628 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.759592 2562 generic.go:356] "Generic (PLEG): container finished" podID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" containerID="3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd" exitCode=0 Jul 06 20:36:46.759831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.759653 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" Jul 06 20:36:46.759831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.759683 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" event={"ID":"053402bc-beaf-4ec3-9ce5-2dc0978fff87","Type":"ContainerDied","Data":"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd"} Jul 06 20:36:46.759831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.759725 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f48df4fb-qfbvj" event={"ID":"053402bc-beaf-4ec3-9ce5-2dc0978fff87","Type":"ContainerDied","Data":"1bd329b6e81f590cfda0bd806ae05c32709f5594b5484cd6fe6bc8acc639308b"} Jul 06 20:36:46.759831 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.759742 2562 scope.go:117] "RemoveContainer" containerID="3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd" Jul 06 20:36:46.769257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.769235 2562 scope.go:117] "RemoveContainer" containerID="3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd" Jul 06 20:36:46.769573 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:36:46.769553 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd\": container with ID starting with 3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd not found: ID does not exist" containerID="3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd" Jul 06 20:36:46.769634 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.769585 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd"} err="failed to get container status \"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd\": rpc error: code = NotFound desc = could not find container \"3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd\": container with ID starting with 3100915a92b065cf13a352f60acd4b0b97104b0b6077c334b48869b170d319cd not found: ID does not exist" Jul 06 20:36:46.793102 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.793061 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:36:46.802277 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:46.802248 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f48df4fb-qfbvj"] Jul 06 20:36:48.560012 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:36:48.559975 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" path="/var/lib/kubelet/pods/053402bc-beaf-4ec3-9ce5-2dc0978fff87/volumes" Jul 06 20:37:57.352189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.352071 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:37:57.352809 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.352295 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-86785d597b-wgxjh" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerName="manager" containerID="cri-o://fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c" gracePeriod=30 Jul 06 20:37:57.611457 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.611388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:37:57.708879 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.708839 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert\") pod \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " Jul 06 20:37:57.709085 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.708919 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca\") pod \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " Jul 06 20:37:57.709085 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.708979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zpz2\" (UniqueName: \"kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2\") pod \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\" (UID: \"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8\") " Jul 06 20:37:57.709334 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.709311 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" (UID: "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:57.711026 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.710992 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" (UID: "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:57.711195 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.711020 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2" (OuterVolumeSpecName: "kube-api-access-9zpz2") pod "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" (UID: "4d2f6682-62bf-433d-be6a-f6a33d1eb4d8"). InnerVolumeSpecName "kube-api-access-9zpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:57.810245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.810204 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpz2\" (UniqueName: \"kubernetes.io/projected/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-kube-api-access-9zpz2\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:37:57.810245 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.810241 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-webhook-cert\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:37:57.810452 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:57.810254 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8-service-ca\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:37:58.209803 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.209758 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerID="fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c" exitCode=0 Jul 06 20:37:58.209803 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.209809 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-wgxjh" Jul 06 20:37:58.210057 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.209839 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-wgxjh" event={"ID":"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8","Type":"ContainerDied","Data":"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c"} Jul 06 20:37:58.210057 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.209895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-wgxjh" event={"ID":"4d2f6682-62bf-433d-be6a-f6a33d1eb4d8","Type":"ContainerDied","Data":"f29d9c2891b05e6a0cd08349c1599d1c7608a588da766dd22a0f28f99254cdcc"} Jul 06 20:37:58.210057 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.209919 2562 scope.go:117] "RemoveContainer" containerID="fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c" Jul 06 20:37:58.220284 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.220262 2562 scope.go:117] "RemoveContainer" containerID="fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c" Jul 06 20:37:58.220526 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:37:58.220506 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c\": container with ID starting with fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c not found: ID does not exist" containerID="fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c" Jul 06 20:37:58.220592 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.220542 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c"} err="failed to get container status \"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c\": rpc error: code = NotFound desc = could not find container \"fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c\": container with ID starting with fec31dfaca9d464459bf5a521b01eada1779d49c5678d6e9e6f53398615bd67c not found: ID does not exist" Jul 06 20:37:58.237992 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.237963 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:37:58.242217 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.242179 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86785d597b-wgxjh"] Jul 06 20:37:58.560305 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.560278 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" path="/var/lib/kubelet/pods/4d2f6682-62bf-433d-be6a-f6a33d1eb4d8/volumes" Jul 06 20:37:58.596928 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.596881 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-86785d597b-wgxjh" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.60:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:37:58.999921 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:58.999806 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86785d597b-5gz9p"] Jul 06 20:37:59.001230 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001203 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" containerName="authorino" Jul 06 20:37:59.001439 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001413 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" containerName="authorino" Jul 06 20:37:59.001554 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001531 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerName="manager" Jul 06 20:37:59.001614 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001565 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerName="manager" Jul 06 20:37:59.002553 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001783 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2f6682-62bf-433d-be6a-f6a33d1eb4d8" containerName="manager" Jul 06 20:37:59.002553 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.001808 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="053402bc-beaf-4ec3-9ce5-2dc0978fff87" containerName="authorino" Jul 06 20:37:59.007807 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.007777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.009983 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.009953 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86785d597b-5gz9p"] Jul 06 20:37:59.011081 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.011057 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sdpss\"" Jul 06 20:37:59.011081 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.011072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:37:59.011289 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.011083 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:37:59.122511 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.122476 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwgdt\" (UniqueName: \"kubernetes.io/projected/922752c2-f66a-47d6-9a79-8e1726ba061d-kube-api-access-xwgdt\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.122511 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.122515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/922752c2-f66a-47d6-9a79-8e1726ba061d-webhook-cert\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.122746 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.122625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/922752c2-f66a-47d6-9a79-8e1726ba061d-service-ca\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.223922 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.223876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwgdt\" (UniqueName: \"kubernetes.io/projected/922752c2-f66a-47d6-9a79-8e1726ba061d-kube-api-access-xwgdt\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.223922 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.223922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/922752c2-f66a-47d6-9a79-8e1726ba061d-webhook-cert\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.224259 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.224004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/922752c2-f66a-47d6-9a79-8e1726ba061d-service-ca\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.224798 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.224776 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/922752c2-f66a-47d6-9a79-8e1726ba061d-service-ca\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.226480 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.226461 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/922752c2-f66a-47d6-9a79-8e1726ba061d-webhook-cert\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.233554 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.233520 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwgdt\" (UniqueName: \"kubernetes.io/projected/922752c2-f66a-47d6-9a79-8e1726ba061d-kube-api-access-xwgdt\") pod \"maas-controller-86785d597b-5gz9p\" (UID: \"922752c2-f66a-47d6-9a79-8e1726ba061d\") " pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.321518 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.321482 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:37:59.471317 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:37:59.471281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86785d597b-5gz9p"] Jul 06 20:37:59.473058 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:37:59.473028 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922752c2_f66a_47d6_9a79_8e1726ba061d.slice/crio-32422c6d359bc8a484189a4b6388948f7b40a0d44cc1ee06abb77a161227d449 WatchSource:0}: Error finding container 32422c6d359bc8a484189a4b6388948f7b40a0d44cc1ee06abb77a161227d449: Status 404 returned error can't find the container with id 32422c6d359bc8a484189a4b6388948f7b40a0d44cc1ee06abb77a161227d449 Jul 06 20:38:00.226621 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:38:00.226532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-5gz9p" event={"ID":"922752c2-f66a-47d6-9a79-8e1726ba061d","Type":"ContainerStarted","Data":"c32ef8b10e336c670ddb8dddef71b47e6d31a90be65fcb4f2efa5d8304e06b73"} Jul 06 20:38:00.226621 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:38:00.226595 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:38:00.226621 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:38:00.226615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-5gz9p" event={"ID":"922752c2-f66a-47d6-9a79-8e1726ba061d","Type":"ContainerStarted","Data":"32422c6d359bc8a484189a4b6388948f7b40a0d44cc1ee06abb77a161227d449"} Jul 06 20:38:00.247878 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:38:00.247827 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86785d597b-5gz9p" podStartSLOduration=1.751429699 podStartE2EDuration="2.247809884s" totalImagesPullingTime="496.380185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:00.245747152 +0000 UTC m=+822.786803589" watchObservedRunningTime="2026-07-06 20:38:00.247809884 +0000 UTC m=+822.788866320" Jul 06 20:38:11.238189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:38:11.238131 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:39:18.490317 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:39:18.490229 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:39:18.492299 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:39:18.492276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:42:02.234038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:02.233985 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-86785d597b-5gz9p" podUID="922752c2-f66a-47d6-9a79-8e1726ba061d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.65:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:42:06.729798 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.729764 2562 generic.go:356] "Generic (PLEG): container finished" podID="b1b9a371-da48-4c41-99b3-c77decdd73e6" containerID="49630aa752878101a57b0dc11a2bcd288f707769beac702c247cc417aa222902" exitCode=1 Jul 06 20:42:06.730273 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.729845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" event={"ID":"b1b9a371-da48-4c41-99b3-c77decdd73e6","Type":"ContainerDied","Data":"49630aa752878101a57b0dc11a2bcd288f707769beac702c247cc417aa222902"} Jul 06 20:42:06.730346 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.730279 2562 scope.go:117] "RemoveContainer" containerID="49630aa752878101a57b0dc11a2bcd288f707769beac702c247cc417aa222902" Jul 06 20:42:06.730640 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.730626 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:42:06.731522 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.731463 2562 generic.go:356] "Generic (PLEG): container finished" podID="922752c2-f66a-47d6-9a79-8e1726ba061d" containerID="c32ef8b10e336c670ddb8dddef71b47e6d31a90be65fcb4f2efa5d8304e06b73" exitCode=1 Jul 06 20:42:06.731583 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.731546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-5gz9p" event={"ID":"922752c2-f66a-47d6-9a79-8e1726ba061d","Type":"ContainerDied","Data":"c32ef8b10e336c670ddb8dddef71b47e6d31a90be65fcb4f2efa5d8304e06b73"} Jul 06 20:42:06.731876 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:06.731862 2562 scope.go:117] "RemoveContainer" containerID="c32ef8b10e336c670ddb8dddef71b47e6d31a90be65fcb4f2efa5d8304e06b73" Jul 06 20:42:07.405361 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:07.405255 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:42:07.739478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:07.739387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" event={"ID":"b1b9a371-da48-4c41-99b3-c77decdd73e6","Type":"ContainerStarted","Data":"d7991a34a5c4b42dea8822172102b8f75240553e0499a7b8c03752bc5938adcc"} Jul 06 20:42:07.739478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:07.739470 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:42:07.741263 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:07.741238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86785d597b-5gz9p" event={"ID":"922752c2-f66a-47d6-9a79-8e1726ba061d","Type":"ContainerStarted","Data":"fbd4017bef708be4413d22fb4279fa645e38b3eec1e25d2cebd4c3e8d1cf701b"} Jul 06 20:42:07.741478 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:07.741443 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:42:18.755601 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:18.755567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86785d597b-5gz9p" Jul 06 20:42:38.753066 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:42:38.753034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pgn22" Jul 06 20:43:02.572615 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:02.572569 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:43:02.573028 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:02.572806 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" podUID="22f1b357-e4c9-4366-a72f-cc3751672124" containerName="manager" containerID="cri-o://ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee" gracePeriod=10 Jul 06 20:43:06.643372 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.643340 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:43:06.743544 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.743462 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jzb4\" (UniqueName: \"kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4\") pod \"22f1b357-e4c9-4366-a72f-cc3751672124\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " Jul 06 20:43:06.743544 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.743497 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume\") pod \"22f1b357-e4c9-4366-a72f-cc3751672124\" (UID: \"22f1b357-e4c9-4366-a72f-cc3751672124\") " Jul 06 20:43:06.743893 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.743868 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "22f1b357-e4c9-4366-a72f-cc3751672124" (UID: "22f1b357-e4c9-4366-a72f-cc3751672124"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:43:06.745510 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.745489 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4" (OuterVolumeSpecName: "kube-api-access-7jzb4") pod "22f1b357-e4c9-4366-a72f-cc3751672124" (UID: "22f1b357-e4c9-4366-a72f-cc3751672124"). InnerVolumeSpecName "kube-api-access-7jzb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:43:06.845115 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.845075 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzb4\" (UniqueName: \"kubernetes.io/projected/22f1b357-e4c9-4366-a72f-cc3751672124-kube-api-access-7jzb4\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:43:06.845115 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:06.845108 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22f1b357-e4c9-4366-a72f-cc3751672124-extensions-socket-volume\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:43:07.104429 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.104389 2562 generic.go:356] "Generic (PLEG): container finished" podID="22f1b357-e4c9-4366-a72f-cc3751672124" containerID="ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee" exitCode=0 Jul 06 20:43:07.104627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.104438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" Jul 06 20:43:07.104627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.104450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" event={"ID":"22f1b357-e4c9-4366-a72f-cc3751672124","Type":"ContainerDied","Data":"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee"} Jul 06 20:43:07.104627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.104504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn" event={"ID":"22f1b357-e4c9-4366-a72f-cc3751672124","Type":"ContainerDied","Data":"1c68d12e5723e620a31fbe568330e31a8e5c522569c6e7b00055d0c59fd45f50"} Jul 06 20:43:07.104627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.104535 2562 scope.go:117] "RemoveContainer" containerID="ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee" Jul 06 20:43:07.115123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.115105 2562 scope.go:117] "RemoveContainer" containerID="ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee" Jul 06 20:43:07.115419 ip-10-0-134-98 kubenswrapper[2562]: E0706 20:43:07.115399 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee\": container with ID starting with ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee not found: ID does not exist" containerID="ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee" Jul 06 20:43:07.115487 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.115427 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee"} err="failed to get container status \"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee\": rpc error: code = NotFound desc = could not find container \"ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee\": container with ID starting with ed3ded1187ab67862c584b04683f9c78881b04dae7625476e718700f7badfcee not found: ID does not exist" Jul 06 20:43:07.134203 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.134175 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:43:07.146957 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:07.146933 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cqstn"] Jul 06 20:43:08.559495 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:43:08.559460 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f1b357-e4c9-4366-a72f-cc3751672124" path="/var/lib/kubelet/pods/22f1b357-e4c9-4366-a72f-cc3751672124/volumes" Jul 06 20:44:03.855289 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.855258 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht"] Jul 06 20:44:03.855766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.855695 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f1b357-e4c9-4366-a72f-cc3751672124" containerName="manager" Jul 06 20:44:03.855766 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.855706 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f1b357-e4c9-4366-a72f-cc3751672124" containerName="manager" Jul 06 20:44:03.855842 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.855768 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f1b357-e4c9-4366-a72f-cc3751672124" containerName="manager" Jul 06 20:44:03.859057 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.859040 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:03.862048 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.862028 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qtc9k\"" Jul 06 20:44:03.876838 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.876808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht"] Jul 06 20:44:03.992188 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.992130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtgf\" (UniqueName: \"kubernetes.io/projected/5adad4e1-40af-4543-8351-360a0aff17ae-kube-api-access-qxtgf\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:03.992367 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:03.992239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adad4e1-40af-4543-8351-360a0aff17ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.092972 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.092933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adad4e1-40af-4543-8351-360a0aff17ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.093122 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.093061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtgf\" (UniqueName: \"kubernetes.io/projected/5adad4e1-40af-4543-8351-360a0aff17ae-kube-api-access-qxtgf\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.093441 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.093420 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adad4e1-40af-4543-8351-360a0aff17ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.104588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.104557 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtgf\" (UniqueName: \"kubernetes.io/projected/5adad4e1-40af-4543-8351-360a0aff17ae-kube-api-access-qxtgf\") pod \"kuadrant-operator-controller-manager-86878c769f-8nkht\" (UID: \"5adad4e1-40af-4543-8351-360a0aff17ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.170273 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.170181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.361327 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.361301 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht"] Jul 06 20:44:04.363472 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:44:04.363442 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adad4e1_40af_4543_8351_360a0aff17ae.slice/crio-892ca38d4cdead2183eda0b35522fb72b2273e2a9b710a5fa77d80660f655ed7 WatchSource:0}: Error finding container 892ca38d4cdead2183eda0b35522fb72b2273e2a9b710a5fa77d80660f655ed7: Status 404 returned error can't find the container with id 892ca38d4cdead2183eda0b35522fb72b2273e2a9b710a5fa77d80660f655ed7 Jul 06 20:44:04.440483 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.440448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" event={"ID":"5adad4e1-40af-4543-8351-360a0aff17ae","Type":"ContainerStarted","Data":"4500c9c6ab5dd1685e223f78eb950c871320eb3f64de602b6af14c7a48fa55fb"} Jul 06 20:44:04.440621 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.440511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" event={"ID":"5adad4e1-40af-4543-8351-360a0aff17ae","Type":"ContainerStarted","Data":"892ca38d4cdead2183eda0b35522fb72b2273e2a9b710a5fa77d80660f655ed7"} Jul 06 20:44:04.440621 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.440599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:04.485932 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:04.485880 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" podStartSLOduration=1.485863195 podStartE2EDuration="1.485863195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:04.480360206 +0000 UTC m=+1187.021416643" watchObservedRunningTime="2026-07-06 20:44:04.485863195 +0000 UTC m=+1187.026919633" Jul 06 20:44:15.455948 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:15.455914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8nkht" Jul 06 20:44:18.530471 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:18.530439 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:44:18.534623 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:44:18.534603 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:45:00.165447 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.165408 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722845-jgbpj"] Jul 06 20:45:00.168933 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.168916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:00.178975 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.178947 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722845-jgbpj"] Jul 06 20:45:00.248431 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.248389 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nf9\" (UniqueName: \"kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9\") pod \"maas-api-key-cleanup-29722845-jgbpj\" (UID: \"43c5b364-adce-4ee1-bfaa-d503ec99267f\") " pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:00.349137 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.349090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nf9\" (UniqueName: \"kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9\") pod \"maas-api-key-cleanup-29722845-jgbpj\" (UID: \"43c5b364-adce-4ee1-bfaa-d503ec99267f\") " pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:00.357709 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.357671 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nf9\" (UniqueName: \"kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9\") pod \"maas-api-key-cleanup-29722845-jgbpj\" (UID: \"43c5b364-adce-4ee1-bfaa-d503ec99267f\") " pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:00.480738 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.480643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:00.628106 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.628080 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722845-jgbpj"] Jul 06 20:45:00.630032 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:45:00.630003 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c5b364_adce_4ee1_bfaa_d503ec99267f.slice/crio-4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf WatchSource:0}: Error finding container 4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf: Status 404 returned error can't find the container with id 4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf Jul 06 20:45:00.787858 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:00.787765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" event={"ID":"43c5b364-adce-4ee1-bfaa-d503ec99267f","Type":"ContainerStarted","Data":"4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf"} Jul 06 20:45:03.813642 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:03.813610 2562 generic.go:356] "Generic (PLEG): container finished" podID="43c5b364-adce-4ee1-bfaa-d503ec99267f" containerID="82a0bcb2bf6c1a83f38a4aad8a78811abe42295ab97852d0218bdd368c3adc76" exitCode=0 Jul 06 20:45:03.814068 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:03.813686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" event={"ID":"43c5b364-adce-4ee1-bfaa-d503ec99267f","Type":"ContainerDied","Data":"82a0bcb2bf6c1a83f38a4aad8a78811abe42295ab97852d0218bdd368c3adc76"} Jul 06 20:45:04.968416 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:04.968387 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:04.994421 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:04.994390 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5nf9\" (UniqueName: \"kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9\") pod \"43c5b364-adce-4ee1-bfaa-d503ec99267f\" (UID: \"43c5b364-adce-4ee1-bfaa-d503ec99267f\") " Jul 06 20:45:04.996524 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:04.996487 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9" (OuterVolumeSpecName: "kube-api-access-n5nf9") pod "43c5b364-adce-4ee1-bfaa-d503ec99267f" (UID: "43c5b364-adce-4ee1-bfaa-d503ec99267f"). InnerVolumeSpecName "kube-api-access-n5nf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:45:05.095889 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:05.095801 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nf9\" (UniqueName: \"kubernetes.io/projected/43c5b364-adce-4ee1-bfaa-d503ec99267f-kube-api-access-n5nf9\") on node \"ip-10-0-134-98.ec2.internal\" DevicePath \"\"" Jul 06 20:45:05.827290 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:05.827252 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" event={"ID":"43c5b364-adce-4ee1-bfaa-d503ec99267f","Type":"ContainerDied","Data":"4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf"} Jul 06 20:45:05.827290 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:05.827277 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722845-jgbpj" Jul 06 20:45:05.827523 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:05.827307 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da991738fa31d9a4729fa39753009d77e16221a8e4b0cbd71e9af4ee65d88cf" Jul 06 20:45:11.865572 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:11.865540 2562 generic.go:356] "Generic (PLEG): container finished" podID="3b04bd76-df05-420e-b00d-1625dea722cb" containerID="625fafed6cc0d32963712d21f764b30fa4f27b23f98eaa9b156f5c63a1038141" exitCode=1 Jul 06 20:45:11.865977 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:11.865611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" event={"ID":"3b04bd76-df05-420e-b00d-1625dea722cb","Type":"ContainerDied","Data":"625fafed6cc0d32963712d21f764b30fa4f27b23f98eaa9b156f5c63a1038141"} Jul 06 20:45:11.866025 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:11.866004 2562 scope.go:117] "RemoveContainer" containerID="625fafed6cc0d32963712d21f764b30fa4f27b23f98eaa9b156f5c63a1038141" Jul 06 20:45:12.873488 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:12.873452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" event={"ID":"3b04bd76-df05-420e-b00d-1625dea722cb","Type":"ContainerStarted","Data":"8e89b6480eb1da45ec69c6ded7e49a8caa27c9256ea31163ca5675f989d03b57"} Jul 06 20:45:12.873987 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:12.873671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:45:23.881731 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:45:23.881699 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qwmwr" Jul 06 20:48:51.504574 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:51.504500 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pgn22_b1b9a371-da48-4c41-99b3-c77decdd73e6/manager/1.log" Jul 06 20:48:51.625049 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:51.625014 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tgclp_10660b83-85b5-4291-8cd5-ef588ed0898a/manager/0.log" Jul 06 20:48:51.746642 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:51.746601 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7f47dc944b-mvpj9_0a035d24-0aa4-4de0-9a86-3389f20d6e64/maas-api/0.log" Jul 06 20:48:51.856848 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:51.856821 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722845-jgbpj_43c5b364-adce-4ee1-bfaa-d503ec99267f/cleanup/0.log" Jul 06 20:48:51.971865 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:51.971828 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86785d597b-5gz9p_922752c2-f66a-47d6-9a79-8e1726ba061d/manager/1.log" Jul 06 20:48:52.083345 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:52.083316 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vwcmz_441b480e-8082-482a-a928-e1260cde0e68/server/0.log" Jul 06 20:48:52.195167 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:52.195071 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qwmwr_3b04bd76-df05-420e-b00d-1625dea722cb/manager/1.log" Jul 06 20:48:52.306783 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:52.306755 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f68bd595-brq2b_b0812b4d-daff-4ea7-8bf3-236ab9611fc6/copy-manifests/0.log" Jul 06 20:48:52.314201 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:52.314180 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f68bd595-brq2b_b0812b4d-daff-4ea7-8bf3-236ab9611fc6/manager/0.log" Jul 06 20:48:53.452277 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.452246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/util/0.log" Jul 06 20:48:53.461968 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.461950 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/pull/0.log" Jul 06 20:48:53.468170 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.468134 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/extract/0.log" Jul 06 20:48:53.590779 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.590744 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/util/0.log" Jul 06 20:48:53.597172 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.597133 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/pull/0.log" Jul 06 20:48:53.603553 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.603535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/extract/0.log" Jul 06 20:48:53.710979 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.710897 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/util/0.log" Jul 06 20:48:53.717440 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.717408 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/pull/0.log" Jul 06 20:48:53.724845 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.724825 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/extract/0.log" Jul 06 20:48:53.839643 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.839608 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/util/0.log" Jul 06 20:48:53.847417 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.847391 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/pull/0.log" Jul 06 20:48:53.854049 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:53.854026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/extract/0.log" Jul 06 20:48:54.102406 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:54.102379 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v2tgk_deaa7ac1-a677-4ae7-8651-c26c8b12ab15/manager/0.log" Jul 06 20:48:54.211123 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:54.211093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-l4nb9_ab8a2440-71eb-45d3-8399-d42a58abdc15/manager/0.log" Jul 06 20:48:54.320512 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:54.320483 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-znt5t_8bad6ecf-539b-4f93-826e-1e45aa557187/kuadrant-console-plugin/0.log" Jul 06 20:48:54.437847 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:54.437754 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tjz7t_91892e9d-38a8-42ff-a919-dfdb29cee95f/registry-server/0.log" Jul 06 20:48:54.566540 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:54.566512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8nkht_5adad4e1-40af-4543-8351-360a0aff17ae/manager/0.log" Jul 06 20:48:55.142916 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:55.142879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ghw29_e5d89105-22b2-4326-b631-6ae10d8f7c8c/istio-proxy/0.log" Jul 06 20:48:55.369862 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:55.369834 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd668b4d4-5v95f_eb612596-dbed-411b-8062-510383d80c94/kube-auth-proxy/0.log" Jul 06 20:48:55.599143 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:55.599110 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/1.log" Jul 06 20:48:55.712827 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:55.712795 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xtq9t_8a5b0f6c-21ab-43ac-a5af-591fe8c6292a/payload-pre-processing/0.log" Jul 06 20:48:56.411647 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:56.411609 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fhbfc_d22f80af-fe78-4286-a15d-8fb510825835/main/0.log" Jul 06 20:48:56.750993 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:48:56.750909 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lrjnz_7265a401-2fa6-4319-832e-b8bef7895592/main/0.log" Jul 06 20:49:01.507103 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.507069 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xk9jl/must-gather-6zwm5"] Jul 06 20:49:01.507565 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.507548 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c5b364-adce-4ee1-bfaa-d503ec99267f" containerName="cleanup" Jul 06 20:49:01.507620 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.507569 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c5b364-adce-4ee1-bfaa-d503ec99267f" containerName="cleanup" Jul 06 20:49:01.507660 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.507654 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c5b364-adce-4ee1-bfaa-d503ec99267f" containerName="cleanup" Jul 06 20:49:01.510910 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.510891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.513439 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.513417 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xk9jl\"/\"default-dockercfg-8nn8l\"" Jul 06 20:49:01.513439 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.513431 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xk9jl\"/\"kube-root-ca.crt\"" Jul 06 20:49:01.513588 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.513522 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xk9jl\"/\"openshift-service-ca.crt\"" Jul 06 20:49:01.515997 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.515978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/810fc426-469d-4d32-92ba-8cc623524a5f-must-gather-output\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.516086 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.516020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxt5\" (UniqueName: \"kubernetes.io/projected/810fc426-469d-4d32-92ba-8cc623524a5f-kube-api-access-wgxt5\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.530397 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.530367 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk9jl/must-gather-6zwm5"] Jul 06 20:49:01.616716 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.616673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/810fc426-469d-4d32-92ba-8cc623524a5f-must-gather-output\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.616921 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.616741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxt5\" (UniqueName: \"kubernetes.io/projected/810fc426-469d-4d32-92ba-8cc623524a5f-kube-api-access-wgxt5\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.617038 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.617018 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/810fc426-469d-4d32-92ba-8cc623524a5f-must-gather-output\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.630341 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.630311 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxt5\" (UniqueName: \"kubernetes.io/projected/810fc426-469d-4d32-92ba-8cc623524a5f-kube-api-access-wgxt5\") pod \"must-gather-6zwm5\" (UID: \"810fc426-469d-4d32-92ba-8cc623524a5f\") " pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.821869 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.821836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" Jul 06 20:49:01.966507 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.966466 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk9jl/must-gather-6zwm5"] Jul 06 20:49:01.967620 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:49:01.967591 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod810fc426_469d_4d32_92ba_8cc623524a5f.slice/crio-9bb4d7905fe9b897983dc9678c4c0a95ebb58ddcb009a920b0339f4280023e54 WatchSource:0}: Error finding container 9bb4d7905fe9b897983dc9678c4c0a95ebb58ddcb009a920b0339f4280023e54: Status 404 returned error can't find the container with id 9bb4d7905fe9b897983dc9678c4c0a95ebb58ddcb009a920b0339f4280023e54 Jul 06 20:49:01.969726 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:01.969712 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:49:02.277720 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:02.277626 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" event={"ID":"810fc426-469d-4d32-92ba-8cc623524a5f","Type":"ContainerStarted","Data":"9bb4d7905fe9b897983dc9678c4c0a95ebb58ddcb009a920b0339f4280023e54"} Jul 06 20:49:03.285986 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:03.285951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" event={"ID":"810fc426-469d-4d32-92ba-8cc623524a5f","Type":"ContainerStarted","Data":"545f4a3c9617debfa6c5cba1ff52130a7d2cb2f2f9569a6a9259e1144f8fcde6"} Jul 06 20:49:03.286413 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:03.285996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" event={"ID":"810fc426-469d-4d32-92ba-8cc623524a5f","Type":"ContainerStarted","Data":"a921269b0dfa0835ae4bc6e4bd3b976b06fd13b1181a9bb5017cb17202abbe77"} Jul 06 20:49:03.306285 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:03.306227 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xk9jl/must-gather-6zwm5" podStartSLOduration=1.4699667889999999 podStartE2EDuration="2.306209329s" totalImagesPullingTime="836.24254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:03.302466033 +0000 UTC m=+1485.843522473" watchObservedRunningTime="2026-07-06 20:49:03.306209329 +0000 UTC m=+1485.847265766" Jul 06 20:49:04.549299 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:04.549256 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z2l27_49f96f3b-f7be-4967-a67a-6b3952922001/global-pull-secret-syncer/0.log" Jul 06 20:49:04.655250 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:04.655213 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k4xpp_babd5803-94f9-49a1-8814-a60283349a51/konnectivity-agent/0.log" Jul 06 20:49:04.706048 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:04.706011 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-98.ec2.internal_100977fbe6cbefbba384bf08f4769a9b/haproxy/0.log" Jul 06 20:49:09.139207 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.139141 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/extract/0.log" Jul 06 20:49:09.168337 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.168301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/util/0.log" Jul 06 20:49:09.202494 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.202467 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggsqt_e129a427-1610-4c09-9958-bc18174d039b/pull/0.log" Jul 06 20:49:09.237730 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.237696 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/extract/0.log" Jul 06 20:49:09.263759 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.263715 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/util/0.log" Jul 06 20:49:09.294587 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.294544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ccfxl_0fe2c00d-1094-48cf-93b6-1118d47c319f/pull/0.log" Jul 06 20:49:09.331413 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.331381 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/extract/0.log" Jul 06 20:49:09.361863 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.361834 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/util/0.log" Jul 06 20:49:09.390197 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.390084 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hh4mt_9a7f962e-e91f-4b5b-85fd-c8f9584891e6/pull/0.log" Jul 06 20:49:09.427329 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.427300 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/extract/0.log" Jul 06 20:49:09.456990 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.456881 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/util/0.log" Jul 06 20:49:09.483303 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.483276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fzdzt_89982a5e-57bc-49be-a129-c58a1f1033cc/pull/0.log" Jul 06 20:49:09.735942 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.735817 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v2tgk_deaa7ac1-a677-4ae7-8651-c26c8b12ab15/manager/0.log" Jul 06 20:49:09.768944 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.768912 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-l4nb9_ab8a2440-71eb-45d3-8399-d42a58abdc15/manager/0.log" Jul 06 20:49:09.799533 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.799496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-znt5t_8bad6ecf-539b-4f93-826e-1e45aa557187/kuadrant-console-plugin/0.log" Jul 06 20:49:09.849582 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.849445 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tjz7t_91892e9d-38a8-42ff-a919-dfdb29cee95f/registry-server/0.log" Jul 06 20:49:09.907278 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:09.907131 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8nkht_5adad4e1-40af-4543-8351-360a0aff17ae/manager/0.log" Jul 06 20:49:11.708189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:11.708139 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-6sc6l_f9c8d0b5-f397-4581-b6bc-16947be0e6c1/cluster-monitoring-operator/0.log" Jul 06 20:49:11.944996 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:11.944961 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5l2ht_d31fefe3-4d3c-4109-aa75-7ca9818d6e4b/node-exporter/0.log" Jul 06 20:49:11.994324 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:11.994272 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5l2ht_d31fefe3-4d3c-4109-aa75-7ca9818d6e4b/kube-rbac-proxy/0.log" Jul 06 20:49:12.036706 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:12.036674 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5l2ht_d31fefe3-4d3c-4109-aa75-7ca9818d6e4b/init-textfile/0.log" Jul 06 20:49:12.469459 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:12.469419 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cnn6n_06e67839-19c2-47fc-92d6-10f8a7a517cc/kube-rbac-proxy-main/0.log" Jul 06 20:49:12.512458 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:12.512408 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cnn6n_06e67839-19c2-47fc-92d6-10f8a7a517cc/kube-rbac-proxy-self/0.log" Jul 06 20:49:12.549251 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:12.549210 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cnn6n_06e67839-19c2-47fc-92d6-10f8a7a517cc/openshift-state-metrics/0.log" Jul 06 20:49:12.959431 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:12.959385 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-7x4mm_afc47ba1-4245-4770-a5d5-562da03326aa/prometheus-operator/0.log" Jul 06 20:49:13.003883 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:13.003847 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-7x4mm_afc47ba1-4245-4770-a5d5-562da03326aa/kube-rbac-proxy/0.log" Jul 06 20:49:13.054085 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:13.054038 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-296c7_b480046e-7b9f-4a9f-918b-a32b302265c7/prometheus-operator-admission-webhook/0.log" Jul 06 20:49:13.102544 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:13.102515 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db4f9cd89-28xwp_18e53078-35a6-4e84-8051-0713529ec919/telemeter-client/0.log" Jul 06 20:49:13.143661 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:13.143626 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db4f9cd89-28xwp_18e53078-35a6-4e84-8051-0713529ec919/reload/0.log" Jul 06 20:49:13.172749 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:13.172715 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db4f9cd89-28xwp_18e53078-35a6-4e84-8051-0713529ec919/kube-rbac-proxy/0.log" Jul 06 20:49:14.516755 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.516726 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-tkfrb_a51e4603-2179-4622-bed7-8cfdd1a920fa/networking-console-plugin/0.log" Jul 06 20:49:14.724236 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.724189 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls"] Jul 06 20:49:14.732476 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.732440 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.742497 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.742445 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls"] Jul 06 20:49:14.867003 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.866957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-podres\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.867333 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.867307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-proc\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.867479 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.867461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-lib-modules\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.867626 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.867608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnw9\" (UniqueName: \"kubernetes.io/projected/58098129-6eea-4e0e-a012-6f8cc285c43d-kube-api-access-mmnw9\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.867740 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.867726 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-sys\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.968682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-proc\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.968682 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-lib-modules\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.968898 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968787 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-proc\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.968898 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnw9\" (UniqueName: \"kubernetes.io/projected/58098129-6eea-4e0e-a012-6f8cc285c43d-kube-api-access-mmnw9\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.968898 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-sys\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.969005 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968911 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-lib-modules\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.969005 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.968960 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-sys\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.969073 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.969012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-podres\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.969131 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.969117 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58098129-6eea-4e0e-a012-6f8cc285c43d-podres\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:14.981829 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:14.979351 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnw9\" (UniqueName: \"kubernetes.io/projected/58098129-6eea-4e0e-a012-6f8cc285c43d-kube-api-access-mmnw9\") pod \"perf-node-gather-daemonset-hp7ls\" (UID: \"58098129-6eea-4e0e-a012-6f8cc285c43d\") " pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:15.046257 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.046223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:15.178630 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.178598 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:49:15.183466 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.183437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/3.log" Jul 06 20:49:15.231456 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.231423 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls"] Jul 06 20:49:15.232026 ip-10-0-134-98 kubenswrapper[2562]: W0706 20:49:15.232004 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58098129_6eea_4e0e_a012_6f8cc285c43d.slice/crio-a61f4edecb3d42ce4bebf82f0a667488dc51a873b7252a101d1331fcd1373ad5 WatchSource:0}: Error finding container a61f4edecb3d42ce4bebf82f0a667488dc51a873b7252a101d1331fcd1373ad5: Status 404 returned error can't find the container with id a61f4edecb3d42ce4bebf82f0a667488dc51a873b7252a101d1331fcd1373ad5 Jul 06 20:49:15.403819 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.403783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" event={"ID":"58098129-6eea-4e0e-a012-6f8cc285c43d","Type":"ContainerStarted","Data":"d0f52f295fb226dd40bdefb62ff1fc794edd7f3ec6e10e88cda0a7ef306e4f14"} Jul 06 20:49:15.404002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.403839 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" event={"ID":"58098129-6eea-4e0e-a012-6f8cc285c43d","Type":"ContainerStarted","Data":"a61f4edecb3d42ce4bebf82f0a667488dc51a873b7252a101d1331fcd1373ad5"} Jul 06 20:49:15.404002 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.403894 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:15.428611 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.428495 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" podStartSLOduration=1.428474872 podStartE2EDuration="1.428474872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:15.423395752 +0000 UTC m=+1497.964452189" watchObservedRunningTime="2026-07-06 20:49:15.428474872 +0000 UTC m=+1497.969531311" Jul 06 20:49:15.689967 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.689904 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55b7599c8b-m427z_f99b3e69-6565-44c2-bf60-0bc684b9ab5b/console/0.log" Jul 06 20:49:15.726189 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:15.726134 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-s8xlc_38829c8e-b2ee-4f20-a4f4-21d637d443b0/download-server/0.log" Jul 06 20:49:16.284522 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:16.284493 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-85srj_20f76774-8494-48e8-a502-b6cdfc3d6c1d/volume-data-source-validator/0.log" Jul 06 20:49:17.358209 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:17.358179 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4glv9_abb79635-08ac-4b40-93d2-f18435f585e8/dns/0.log" Jul 06 20:49:17.398087 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:17.398049 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4glv9_abb79635-08ac-4b40-93d2-f18435f585e8/kube-rbac-proxy/0.log" Jul 06 20:49:17.638057 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:17.637969 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8np8r_f7083172-62a8-4506-b77e-7112ccde6e98/dns-node-resolver/0.log" Jul 06 20:49:18.261394 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:18.261365 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dcb484c46-mmn49_58e0099c-0f37-42d6-9ac3-181ed259799c/registry/0.log" Jul 06 20:49:18.345095 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:18.345064 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h8rnd_212a155d-c62b-42a3-bbeb-d230b050d072/node-ca/0.log" Jul 06 20:49:18.588424 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:18.588392 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:49:18.590774 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:18.590749 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-hfd54_2d44e299-d22d-49d2-8ab4-b67020497397/console-operator/2.log" Jul 06 20:49:19.389627 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:19.389596 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ghw29_e5d89105-22b2-4326-b631-6ae10d8f7c8c/istio-proxy/0.log" Jul 06 20:49:19.468266 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:19.468221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd668b4d4-5v95f_eb612596-dbed-411b-8062-510383d80c94/kube-auth-proxy/0.log" Jul 06 20:49:19.528908 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:19.528878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/0.log" Jul 06 20:49:19.563755 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:19.563722 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/1.log" Jul 06 20:49:19.593687 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:19.593658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xtq9t_8a5b0f6c-21ab-43ac-a5af-591fe8c6292a/payload-pre-processing/0.log" Jul 06 20:49:20.282649 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:20.282624 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h97f5_6195ed3c-3037-4c3e-85c5-f381fce16615/serve-healthcheck-canary/0.log" Jul 06 20:49:20.888103 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:20.888061 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lxbj_ab0299c6-c4e2-4e1c-885c-a86f76517386/kube-rbac-proxy/0.log" Jul 06 20:49:20.918765 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:20.918724 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lxbj_ab0299c6-c4e2-4e1c-885c-a86f76517386/exporter/0.log" Jul 06 20:49:20.960626 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:20.960583 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8lxbj_ab0299c6-c4e2-4e1c-885c-a86f76517386/extractor/0.log" Jul 06 20:49:21.425101 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:21.425058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xk9jl/perf-node-gather-daemonset-hp7ls" Jul 06 20:49:23.731845 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.731761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pgn22_b1b9a371-da48-4c41-99b3-c77decdd73e6/manager/0.log" Jul 06 20:49:23.733777 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.733750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pgn22_b1b9a371-da48-4c41-99b3-c77decdd73e6/manager/1.log" Jul 06 20:49:23.816018 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.815548 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tgclp_10660b83-85b5-4291-8cd5-ef588ed0898a/manager/0.log" Jul 06 20:49:23.867517 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.867484 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7f47dc944b-mvpj9_0a035d24-0aa4-4de0-9a86-3389f20d6e64/maas-api/0.log" Jul 06 20:49:23.904980 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.904951 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722845-jgbpj_43c5b364-adce-4ee1-bfaa-d503ec99267f/cleanup/0.log" Jul 06 20:49:23.979182 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.979139 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86785d597b-5gz9p_922752c2-f66a-47d6-9a79-8e1726ba061d/manager/0.log" Jul 06 20:49:23.984616 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:23.984549 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86785d597b-5gz9p_922752c2-f66a-47d6-9a79-8e1726ba061d/manager/1.log" Jul 06 20:49:24.059299 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:24.059274 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vwcmz_441b480e-8082-482a-a928-e1260cde0e68/server/0.log" Jul 06 20:49:24.104502 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:24.104472 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qwmwr_3b04bd76-df05-420e-b00d-1625dea722cb/manager/1.log" Jul 06 20:49:24.130559 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:24.130524 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qwmwr_3b04bd76-df05-420e-b00d-1625dea722cb/manager/0.log" Jul 06 20:49:24.164313 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:24.164286 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f68bd595-brq2b_b0812b4d-daff-4ea7-8bf3-236ab9611fc6/manager/0.log" Jul 06 20:49:24.194539 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:24.194508 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f68bd595-brq2b_b0812b4d-daff-4ea7-8bf3-236ab9611fc6/copy-manifests/0.log" Jul 06 20:49:25.600471 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:25.600437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ghw29_e5d89105-22b2-4326-b631-6ae10d8f7c8c/istio-proxy/0.log" Jul 06 20:49:25.662974 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:25.662947 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd668b4d4-5v95f_eb612596-dbed-411b-8062-510383d80c94/kube-auth-proxy/0.log" Jul 06 20:49:25.700678 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:25.700643 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/0.log" Jul 06 20:49:25.737239 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:25.737158 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/1.log" Jul 06 20:49:25.756436 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:25.756387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xtq9t_8a5b0f6c-21ab-43ac-a5af-591fe8c6292a/payload-pre-processing/0.log" Jul 06 20:49:26.937839 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:26.937809 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ghw29_e5d89105-22b2-4326-b631-6ae10d8f7c8c/istio-proxy/0.log" Jul 06 20:49:26.993708 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:26.993665 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd668b4d4-5v95f_eb612596-dbed-411b-8062-510383d80c94/kube-auth-proxy/0.log" Jul 06 20:49:27.033746 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:27.033716 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/0.log" Jul 06 20:49:27.069121 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:27.069088 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cl6m6_d1ff50c0-9de3-4711-9843-e2ed67efcec2/istio-proxy/1.log" Jul 06 20:49:27.113360 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:27.113325 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-xtq9t_8a5b0f6c-21ab-43ac-a5af-591fe8c6292a/payload-pre-processing/0.log" Jul 06 20:49:31.233122 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:31.233037 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-s9nl5_fbb8cc97-039a-4f28-8c92-aaad7bf9e34f/kube-storage-version-migrator-operator/1.log" Jul 06 20:49:31.234113 ip-10-0-134-98 kubenswrapper[2562]: I0706 20:49:31.234078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-s9nl5_fbb8cc97-039a-4f28-8c92-aaad7bf9e34f/kube-storage-version-migrator-operator/0.log"